1/47

9 Factors that Influence Prescribers’ Response to 

Alerts at the Point-of-Care: 

Implications for VA Order Check Design

Alissa L. Russ, PhD

alissa.russ@va.gov 

 

Research Scientist

Roudebush VA HSR&D Center of Excellence on

 Implementing Evidence-Based Practice (CIEBP) 

Regenstrief Institute, Inc.

 

Adjunct Assistant Professor

Purdue University College of Pharmacy
2/47

My Background

• Human Factors Engineering:

– design of technology, processes, and work systems so they 

are compatible with human cognitive and physical 

capabilities and limitations
3/47

Audience Question #1

What is your primary role in the VA?

a. informatics

b. patient care

c. research

d. other 

e. work outside the VA
4/47

Outline

1. Overview of VA alerts

2. Results from field observations and 

 interviews with VA prescribers 

3. Framework for prescriber–alert interaction

4. Describe potential implications for VA alert 

design and medication safety
5/47

Computerized Medication Alerts

Background:

– 1.5 million preventable adverse drug events annually in 

U.S. (IOM, 2006)

Literature: 

• Alert fatigue (Grizzle et al, 2007)

• Database analyses, surveys (Van der Sijs et al, 2006)

Study Aim: 

Identify factors that influence how prescribers perceive and 

respond to alerts at the point-of-care

 
6/47

Medication Alerts (i.e. order checks)
7/47

Medication Alerts (i.e. order checks)
8/47

Medication Alerts (i.e. order checks)
9/47

Select 

medication

Alert(s) 

appear

Initiate 

signature

Alert(s) 

appear 

sent to 

pharmacy

Ordering Process (simplified)
10/47

Audience Question #2:

• According to human factors science, what is the 

primary way that providers learn about how 

health IT is designed and intended to be used for 

patient care?

a. operations manual

b. from their peers

c. training sessions

d. software interface design

e. talk to the developers
11/47

Answer for Question #2:

• According to human factors science, what is the 

primary way that providers learn about how 

health IT is designed and intended to be used for 

patient care?

a. operations manual

b. from their peers

c. training sessions

d. software interface design

e. talk to the developers
12/47

Methods

• Field observations and interviews

– Major Midwestern VA Medical Center

– 1⁄2 day per prescriber

• NPs, MDs, Clinical Pharmacists

• Qualitative analysis

– Inductive, emergent themes (Campbell 2006; Patterson 2002)

– Team consensus

– MAXQDA software
13/47

Data Analysis

Observations

Total hrs 102.8 hrs

Typed

Notes

351 pgs

Patients 146 

Alerts 320 

Qualitative Analysis

Meetings 30 (27.5 hrs)

Jason 

Saleem PhD

Alan Zillich

PharmD

Sue 

McManus

PhD, NP
14/47

Recruiting Results

Prescribers N=30 

Primary Care 20 prescribers:

(PC) 5 PC teams

4 per team

Specialty clinics 10 prescribers

8 clinics

Expertise 18 physicians

 7 nurse practitioners

5 pharmacists

Age 42 (27-63) yrs

Years in VA 10 (<1-24) yrs

Gender 14M/16F 
15/47

Results: prescriber-alert interaction

• 44 themes

• 9 overarching factors

• framework 
16/47

System

Design 

Prescriber

Input

System 

Image 

Prescriber

Perceptions

Programmer’s 

Mental Model 

Prescriber’s 

Mental Model

Underlying Framework 

From Norman 1990 and adapted by Russ et al 2012 for prescribing
17/47

System

Design 

1. Alert System Logic 

2. Alert System Redundancy

Prescriber

Input

System 

Image 

Prescriber

Perceptions

Programmer’s 

Mental Model 

Prescriber’s 

Mental Model

Prescriber-Alert Interaction

Norman 1990 

Russ et al, IJMI 2012 
18/47

1. Alert System Logic

• External Crosschecks:

– Compares VA meds across VAMCs

– Compares VA meds to entered non-VA meds

– Perceived as strength of VA alert system

• Detection (over/under detection)

– Identified gaps; prescribers wanted more alerts for:

• lithium 

• thyroid and liver function

• non-steriodal anti-inflammatory drugs (NSAIDs)

• medications that can affect psychiatry patients
19/47

2. Alert System Redundancy

Repetition within an encounter

• Example:

1: NP orders niacin; alert for niacin/pravastatin

2: NP signs order, alert appears again

3: NP orders pravastatin; alert for pravastatin/niacin

4: NP signs order, alert appears again

– Observer noted: We have now seen the same alert 

4 times in last 10 min or less.
20/47

System

Design 

1. Alert System Logic 

2. Alert System Redundancy

Prescriber

Input

3. Alert Display

4. Alert Content

System 

Image 

Prescriber

Perceptions

Programmer’s 

Mental Model 

Prescriber’s 

Mental Model

Prescriber-Alert Interaction

Norman 1990 

Russ et al, IJMI 2012 
21/47

3. Alert Display

• 5 themes included:

– Format

– Salience

– Timing

– Level of intrusiveness

– Retrievability

• “Redesigning Medication Alerts to Support 

Prescriber Workflow”

– HSR&D grant PPO #09-298 

– Coming soon to a journal near you... 
22/47

4. Alert Content

Specification: explanation of why an alert was triggered

Order check appears:

Duplicate drug class, non- opioids [antitussives]/expectorants. 

Non-VA medication guiafenesin. 

 

MD: “I don’t even know what that means. It 

 says expectorants, but it doesn’t say 

 what the other [medication] is. It says 

 non-VA medication guiafenesin, but 

 you’re [pt 4] getting that here, right?” 
23/47

Audience Question #3

Out of the 30 prescribers in the study, data from ____

prescribers indicated that there was confusion about why 

alerts were triggered. 

a. less than 5

b. 5-9 

c. 10-19

d. more than 20
24/47

Audience Question #3

Out of the 30 prescribers in the study, data from 

 21 prescribers indicated that there was confusion about why 

alerts were triggered. 

a. less than 5

b. 5-9 

c. 10-19

d. more than 20 Implication: a lack of specification poses a 

substantial barrier to resolving alerts; 

additional attention is needed on the clinical 

content presented by alerts
25/47

5. Cognitive Factors 

System

Design 

1. Alert System Logic 

2. Alert System Redundancy

Prescriber

Input

3. Alert Display

4. Alert Content

System 

Image 

Prescriber

Perceptions

Programmer’s 

Mental Model 

Prescriber’s 

Mental Model

Prescriber-Alert Interaction

Norman 1988 

Russ et al, IJMI 2012 
26/47

5. Cognitive factors

 Audience Question #4:

In this study, alerts sometimes supported 

awareness by providing new information for 

prescribers. Prescribers found alerts 

particularly helpful for:

a. new patients

b. new medications

c. medications they rarely prescribed

d. allergies documented by someone else

e. all of the above
27/47

Answer for #4:

In this study, alerts sometimes supported 

awareness by providing new information for 

prescribers. Prescribers found alerts 

particularly helpful for:

a. new patients

b. new medications

c. medications they rarely prescribed

d. allergies documented by someone else

e. all of the above 
28/47

5. Cognitive Factors 

System

Design 

1. Alert System Logic 

2. Alert System Redundancy

Prescriber

Input

3. Alert Display

4. Alert Content 6. Pharmaceutical 

Knowledge

System 

Image 

Prescriber

Perceptions

Programmer’s 

Mental Model 

Prescriber’s 

Mental Model

Prescriber-Alert Interaction

Norman 1990 

Russ et al, IJMI 2012 
29/47

6. Pharmaceutical Knowledge: 

“I talk to the clinical pharmacist to 

resolve order checks. I like having the 

pharmacist in the room here with me.”

“I’ll ask the pharmacist in the clinic, 

‘Is this an important interaction?’ .”

“If I’m not in the room, the doctors don’t 

know what to do with the alert.”
30/47

6. Pharmaceutical Knowledge: 

• Pharmacist consultation and proximity: pharmacists 

consulted in real-time to help resolve alerts

“Physicians are not trained like pharmacists. We have to 

learn what a significant, clinically-relevant interaction 

is.” 

 - VA physician

• Implications:

– Alert language, information currently inadequate

– Universal design needed for alerts to help support prescribers with 

different training and levels of experience
31/47

5. Cognitive Factors 

System

Design 

1. Alert System Logic 

2. Alert System Redundancy

Prescriber

Input

3. Alert Display

4. Alert Content 6. Pharmaceutical 

Knowledge

 7. Medication Management 

 8. Workflow 

System 

Image 

Prescriber

Perceptions

Programmer’s 

Mental Model 

Prescriber’s 

Mental Model

Prescriber-Alert Interaction

Norman 1990 

Russ et al, IJMI 2012 
32/47

7. Medication Management

• Computerized Provider Order Entry (CPOE) design

– CPOE design sometimes hindered alert resolution

– MD orders mycophenolate 

– Duplicate drug class alert: 

 mycophenolate/azathioprine 

– MD overrides alert

– MD explains: “I couldn’t discontinue the [azathioprine] 

because it is coming from [another VAMC]....I can only tell 

the patient to stop it.”

– Implications: need more advanced CPOE and alert for EHR 

interoperability
33/47

8. Workflow

 Audience Question #5: 

In this study, after computer delays reached 

________, prescribers began expressing 

frustration.

a. 10-15 sec

b. 16-30 sec

c. 31-60 sec

d. 61-90 sec
34/47

Answer for #5: 

In this study, after computer delays reached 

________, prescribers began expressing 

frustration.

a. 10-15 sec

b. 16-30 sec

c. 31-60 sec

d. 61-90 sec
35/47

5. Cognitive Factors 

System

Design 

1. Alert System Logic 

2. Alert System Redundancy

Prescriber

Input

3. Alert Display

4. Alert Content 6. Pharmaceutical 

Knowledge

 7. Medication Management 

 8. Workflow 

9. Alert System Reliability 

System 

Image 

Prescriber

Perceptions

Programmer’s 

Mental Model 

Prescriber’s 

Mental Model

Prescriber-Alert Interaction

Norman 1990 

Russ et al, IJMI 2012 
36/47

9. Alert System Reliability

• Common care practices: 

– alerts inappropriately warn against practices that 

apply to broad patient populations
37/47

Alert Trigger

• NPH* insulin/regular 

insulin/metformin

• Mometasone with albuterol

[inhalers].

• Duplicate drug class, 

antiretrovirals 

Prescriber Response

• “There are tons of [diabetic ] 

patients on this combination 

and that is safe.”

• “We have a lot of patients on 

multiple inhalers.”

• “Each patient is on at least 3 

antiretrovirals. The cocktail is 

how we treat patients.”

*Neutral Protamine Hagedorn
38/47

 Common care practices: 

◦ alerts inappropriately warn against practices that 

apply to broad patient populations

Implications:

◦ Reduce alerts that conflict with evidence
39/47

5. Cognitive Factors 

System

Design 

1. Alert System Logic 

2. Alert System Redundancy

Prescriber

Input

3. Alert Display

4. Alert Content 6. Pharmaceutical 

Knowledge

 7. Medication Management 

 8. Workflow 

9. Alert System Reliability 

System 

Image 

Prescriber

Perceptions

Programmer’s 

Mental Model 

Prescriber’s 

Mental Model

Prescriber-Alert Interaction

Norman 1990 

Russ et al, IJMI 2012 
40/47

 Alert interface is often a barrier for 

prescribers, and should be addressed along 

with alert fatigue

 Additional work is needed to understand how 

to present clinical content on alerts, so that 

alerts aid more prescriber types

 Need to prepare alert/CPOE systems for 

increased EHR interoperability
41/47

Conclusions

• One of first studies to examine alerts real-time 

at point-of-care

• Novel framework for prescriber-alert 

interaction 

• Findings may inform alert redesigns to enhance 

patient safety
42/47

“Some are critical interactions....For example, nitrates [and] 

phosphodiesterase inhibitors... 

It has happened before where I didn’t catch this interaction, 

but the computer did.” 

- Physician
43/47
44/47

Funding Support

• U.S. Department of Veterans Affairs, Veterans Health 

Administration

– Health Services Research & Development (HSR&D)

• HSR&D Center of Excellence on Implementing Evidence- Based Practice (CIEBP)

– Center grant #HFP 04-148

• Dr. Russ was supported by a VA HSR&D Associated Health 

Postdoctoral Fellowship

• Drs. Saleem and Zillich were supported by VA HSR&D 

Career Development Awards
45/47

Acknowledgements

• Study Participants

• Project Team

– Jason Saleem, PhD

– Alan Zillich, PharmD

– Sue McManus, NP, PhD

– Brad Doebbeling, MD, MSc 

• Additional thanks:

– Rich Frankel, PhD

– Marshall Jones, PharmD

– Kim Zipper, IT specialist

– David Bickel, IT specialist

– Bruce Molitoris, MD
46/47

Bibliography

• Campbell, E. M., Sittig, D. F., Ash, J. S., Guappone, K. P., & Dykstra, R. H. (2006). Types of unintended 

consequences related to computerized provider order entry. J Am Med Inform Assoc, 13(5), 547-556.

• Grizzle, A. J., Mahmood, M. H., Ko, Y., Murphy, J. E., Armstrong, E. P., Skrepnek, G. H., et al. (2007). Reasons 

provided by prescribers when overriding drug-drug interaction alerts. Am J Manag Care,13(10), 573-578.

• Institute of Medicine of the national academies. (July 2006), Preventing medication errors. Report brief. 

• Norman, D. A. (1990). The Design of Everyday Things (1st ed.). New York, NY: Doubleday.

• Patterson, E. S., Cook, R. I., & Render, M. L. (2002). Improving patient safety by identifying side effects from 

introducing bar coding in medication administration. J Am Med Inform Assoc, 9(5), 540-553.

• Russ AL, Zillich AJ, McManus MS, Doebbeling BD, Saleem JJ. (2012) “Prescribers’ Interactions with 

Medication Alerts at the Point of Prescribing: A Multi-Method, In Situ Investigation of the Human- Computer Interaction”, International Journal of Medical Informatics, 81(4):232-243.

• Van der Sijs, H., Aarts, J., Vulto, A., & Berg, M. (2006). Overriding of drug safety alerts in computerized 

physician order entry. J Am Med Inform Assoc, 13(2), 138-147.
Thank you!

alissa.russ@va.gov