Guideline-Based Decision 

Support for Hypertension with 

ATHENA DSS

Organizational Issues in 

Implementation

Mary K. Goldstein, MD

VA Cyber Seminar, Sept 19, 2006

Views expressed are those of the speaker and not necessarily those of the Department of Veterans 

Affairs or other funding agencies or affiliated institutions
Imagine you have a new 

informatics tool to share...

Interactive Visualization and Exploration of 

Time-oriented Clinical Data Using a 

Distributed Temporal-Abstraction 

Architecture

Yuval Shahar, et al 2003

Available in pubmedcentral
Where to Start?

• You have a cool new tool to improve 

quality of health care, for example, 

– to help clinicians with complex decisions

– to transfer research knowledge into practice 

faster

– to help quality managers analyze clinical data

• The IT tool is designed to integrate with 

the electronic databases/medical record

• How to get started implementing it?
Goals/Objectives of Session

• Overall goal

– to share experience implementing 

information technology (IT) for clinical quality 

improvement (QI)

• Objectives: at end of session, participants 

should be able to...

– consider sociotechnological approach to 

implementing IT in VA health care settings

– identify several key stakeholders 
Perspective

• Physician/health services researcher

• Drawing on expertise of others from wide 

variety of fields (interdisciplinary)

– computer science/medical informatics

– biostats

– sociology

– ...and more
What the Clinician Sees...
ATHENA Hypertension Advisory:

BP- Prescription Graphs

Goldstein, M. K. and B. B. Hoffman (2003). Graphical Displays to Improve GuidelineBased Therapy of Hypertension. Hypertension Primer. J. L. Izzo, Jr and H. R. Black. 

Baltimore, Williams & Wilkins.
What is ATHENA DSS?

• Automated decision support system (DSS) 

– Knowledge-based system automating guidelines

• Built with EON technology for guideline-based decision 

support, developed at Stanford Medical Informatics

– For patients with primary hypertension who meet 

eligibility criteria

• Patient specific information and 

recommendations at the point of care

• Purpose is to improve hypertension control 

and prescription concordance with guidelines

•Athena in Greek mythology is a symbol of good counsel, 

prudent restraint, and practical insight

•Proc AMIA 2000
Sites for Clinical Trial

Palo Alto (in 7 cities), San Francisco, and 

Durham VAMC’s (total 9 separate sites)

San Francisco VA, 

CA Palo Alto VA, 

CA Durham VAMC, 

North Carolina
ATHENA Clients

Advisory

Client

Event

Monitor

Building ATHENA System From 

EON Components

SQL

Patient

Database

ATHENA Clients

EON Servers

Guideline

Interpreter Advisory

Client

Event

Monitor

Temporal

Mediator

VA VISTA VA CPRS

(DHCP)

Data Converter

nightly data 

extraction ATHENA

HTN

Guideline

Knowledge

Base

Protégé ATHENA GUI

Precomputed

Advisories
Server-Client

VA San Francisco 

Clinics

VA Palo Alto Clinics:

•Palo Alto

•Menlo Park

•San Jose

•Monterey

•Livermore

•Stockton

•Modesto

Server in Palo Alto, CA

Durham VA Clinics

Server in Durham, NC
Developing a Model Program

To Provide a Model Program that can be extended to other clinical areas

We selected hypertension as a model for guideline 

implementation because...

• Hypertension is highly prevalent in adult medical practice

• There are excellent evidence-based guidelines for management

• There is also evidence that the guidelines are not well-followed

– a big ‘improvability gap’ in IOM terms

• Steinman, M.A., M.A. Fischer, M.G. Shlipak, H.B. Bosworth, E.Z. Oddone, B.B. 

Hoffman and M.K. Goldstein, Are Clinicians Aware of Their Adherence to 

Hypertension Guidelines? Amer J. Medicine 117:747-54, 2004.
Path to Guideline Adherence

The theoretical model we use for the path to 

guideline adherence is the “Awareness to 

Adherence” model, in which the clinician 

must

– Awareness of guideline

– Acceptance of guideline 

– Adoption of guideline

– Adherence to guideline

Pathman, D. E., T. R. Konard, et al. (1996). "The Awareness-toAdherence Model of the Steps to Clinical Guideline Compliance." 

Medical Care 34:873-889.
Informatics Support for Clinical Practice

Guideline Implementation

Step Facilitators Informatics

Support

Awareness

Priming Activities such as 

profiling of baseline 

performance

Profiling from pharmacy 

and diagnosis database 

Acceptance 

Active education such as 

Academic Detailing; 

Clinical Opinion Leaders

Present evidence 

relevant to patient; allow 

opinion leaders to 

browse knowledge

Adoption

Enabling strategies such as

incorporation into clinic 

workflow

Integration with existing 

EMR

Adherence Reinforcing Strategies such as 

reminders

Point-of-care patientspecific advisories
Challenge of Using IT for Quality 

Improvement

• Technical challenges of using information technology for quality 

improvement (QI)

– Difficult to integrate new forms of decision support into legacy data 

systems and electronic record interfaces 

– We had many design requirements in order to meet research goals 

and institutional goals

– A “sociotechnical” challenge to implement

Goldstein, M., R. Coleman, S. Tu, et. Al. Translating Research Into Practice: 

SocioTechnical Integration of Automated Decision Support for Hypertension in 

Three Medical Centers. JAMIA 11: 368-76, 2004.

Available in pubmedcentral
Decision Support for Common 

Chronic Diseases

The “Field of Dreams” approach to medical 

informatics implementations:

If you build it, they will come

The physician often seen as wondering about a clinical question and 

then seeking out decision support:
Will it Be Used?

• Once decision support is integrated 

technologically, will clinicians use it?

• Many clinical decision support systems are 

used only a tiny percent of time available

• For example, physicians viewed a hyperlipidemia 

guideline only 20 of 2610 visit opportunities (0.8%)

– Maviglia SM, Z.R., Paterno M, Teich JM, Bates DW, Kuperman 

GJ, Automating Complex Guidelines for Chronic Disease: Lessons 

Learned. J Am Med Inform Assoc, 2003. 10: p. 154-165.

– (note that even infrequent use may still be beneficial at very low cost)
“Sociotechnical”* Success

• Technical success

– generates correct recommendations offline

– extracts and uses patient data correctly

– integrates with CPRS to display for the right

• Patient, provider, clinical location, time window

– logs the data needed for research evaluation

• “Sociological” success

– clinicians find it usable and useful

*Berg, M., Patient care information systems and health care work: a 

sociotechnical approach. Int J Med Inf, 1999. 55(2): p. 87-101.

Berg, M., Rationalizing Medical Work: Decision-Support Techniques and 

Medical Practices. Inside Technology, ed. W.E. Bijker, W.B. Carlson, 

and T. Pinch. 1997, Cambridge, Massachusetts: The MIT Press.
Working with Stakeholders

PCPs

IRMS

Clinical Applications 

Coordinators

Admin/

Clinical Mgrs
Some Technical Challenges

• Extracting clinical data from VistA 

• Generating a popup window that appears 

in CPRS

– At the right time, in the right clinic settings, for 

the right clinician, about the right patient

• Logging data about activity in the system

• Security issues

• Maintaining a system that is not on IRMS 

standard priority list
Working with Stakeholders

PCPs

IRMS

Clinical Applications 

Coordinators

Admin/

Clinical Mgrs

programming

networking

Clinic computer

Support staff
Some of the Social Challenges

• Clinicians extremely time-pressured in 

clinic

– Strike balance between ease of access to 

system and ease of ignoring it

• Enormous variability in comfort with 

computers

– And virtually no training time available

• Disagreements about the guidelines

– some want VA GLs, some want JNC
Working with Stakeholders

PCPs

IRMS

Clinical Applications 

Coordinators

Admin/

Clinical Mgrs

programming

networking

Clinic computer

Support staff

PAD

LD

SJ and

VAMC

Stckn

Mod

Durham

San

Francisco
Taking on the Sociotechnical 

Challenge

• Aligning with institutional goals

– Discuss with local stakeholders

– VA performance standards and guidelines

• Speaking the language(s)

– understanding that different computer worlds are worlds apart

• Identify a bridge person to span the gap between IRMS expertise 

and non-VA programmers

• Iterative Design

– With opportunity for re-design cycles after input from key clinical 

staff

– Don’t test in clinic prematurely

• Do your offline testing first

– Test with typical users, not just early adopters

– Recognize need for continual adaptation to our evolving 

informatics infrastructure
Working with Stakeholders

PCPs

IRMS

Clinical Applications 

Coordinators

Admin/

Clinical Mgrs

programming

networking

Clinic computer

Support staff

PAD LD

SJ and

VAMC

Stckn

Mod

ACOS Amb Care

Dep COS for LD

Med Serv Chief

National guideline groups
VA Guidelines
Speaking the Language

• Recruit a VA staff person who is able to talk with 

both IRMS and non-VA programmers

– Who understands VistA file structures

• Recognize that Office of Information has a 

complex and sophisticated process for 

managing projects

– And many competing demands 

• High-level support is important to have but is not 

enough
Understanding the Clinical 

Workflow

• Computer timestamps and clock time

• Conceptualizations of workflow in 

computer systems versus actual workflow

• (see next few slides)

The effects of CPOE on ICU workflow: an observational study. CH Cheng, 

MK Goldstein, E Geller, and RE Levitt. AMIA Annu Symp Proc. 2003; 2003: 

150–154. available in pubmedcentral.
Actual workflow

Computer system

workflow

Reconciliation

Computer system workflow 

diverges from actual workflow
Coordination redundancy:

Entering and interpreting orders

In 97 interruptions of RN to MD, 25% were reminders
Planning the Timeline

• Conceptualization of tasks sequentially

• Develop system

• Test offline for accuracy and usability

• Deploy in production system, limited to users who 

are testing it

• Test in production system (in clinic)

• Go live for clinical trial

• In reality, many tasks have subtasks that 

must be done concurrently with tasks from 

later in sequence
Usability and Usefulness 

Evaluation in Lab Setting

• Martins, S., et al., Evaluation of KNAVE-II: a tool for intelligent query and 

exploration of patient data. Medinfo, 2004. 11(Pt 1): p. 648-652.

a
Evaluation

Flowchart

Patient Data

• Eligibility

•Target BP

•BP under control

• Risk group

• Drug recommendations

• Messages

Evaluation Flowchart

Rules

 

M D Athena

Comparison MD versus ATHENA

Martins SB et al Proc AMIA 2006 in press
“Physician Testers” in Clinical 

Setting

• Project-friendly physicians who test the 

system in early stages in clinic

– Understanding it is not yet complete

– Must be prepared to make changes in 

response to their comments

– Some of these physicians become champions 

for the system

• Include clinical managers in early testing
Consensus Conference Calls

• Knowledge updates required in light of newly published 

clinical trials or new guidelines

– Need a knowledge management process for vetting new 

material and deciding what will be incorporated

– Make this process known to the clinicians who are end-users 

(especially local opinion leaders)

– Invite local input to the discussion

– Encode with a system that allows for easy updating

Goldstein, M.K., B.B. Hoffman, et al, Implementing clinical practice 

guidelines while taking account of changing evidence: ATHENA 

DSS, An easily modifiable decision-support system for managing 

hypertension in primary care. AMIA Symp: 300-4, 2000.
ATHENA Protégé top level
ATHENA Protégé 

GL management

diagram
Eliciting Clinician Feedback

• Clinical Applications Coordinator (CAC) involvement at 

initial launch for large group

• Ongoing monitoring over time*

– Real-time feedback about the patient being seen

• Collected thru the display window

• Must commit to reviewing regularly

• Respond to all comments

• Immediately address problems

*Chan, A., S. Martins, R. Coleman, H. Bosworth, E. Oddone, M. 

Shlipak, S. Tu, M. Musen, B. Hoffman, and M. Goldstein, Post 

Fielding Surveillance of a Guideline-Based Decision Support 

System, in Advances in Patient Safety: From Research to 

Implementation. Vol. 1. Research Findings AHRQ Publication 

Number 05-0021-1, K. Henriksen, et al., Eds. 2005, AHRQ: 

Rockville, MD 20850. p. 331-339.
Adapting to the Evolving IT 

Infrastructure

Example: 

Basis for triggering a popup display window 

Current method:

• CPRS Open Architecture broadcast of CPRS events via 

Windows messaging

– IRMS was going to deactivate this and change to 

CCOW-compliant Context Vault

– We developed a version that works with context vault

– Problem of no user information in Context Vault and 

inconsistent implementation

– Reverting to Windows messaging 
Continuing Challenges 

• No infrastructure support for lab

– scramble from project to project

• Scant funds for development, so doing the work of 

implementation and clinical trial

– need to fund staff through multiple projects

• Funding gap

– National Library of Medicine (NLM) funds new 

informatics (basic science of informatics)

– HSR&D/AHRQ fund implementations for clinical trials 

with patient outcomes

– Who funds all the work in between?
Additional Learning Resources for 

Clinical Decision Support

• Want to learn more about knowledge-based decision 

support?

– Short Course (one afternoon) at Society for Medical Decision 

Making in Boston October 2006

• Want to hear more about a wide variety of clinical 

decision support tools for health professionals and for 

patients?

– Symposium and Workshop at Society for Medical Decision 

Making in Boston October 2006

• AMIA meeting November 2006

– barriers to following guidelines (Lin N et al); offline testing 

(Martins SB et al); CPOE (Zeiger/Johnson et al); decision tool in 

development for use on a patient portal (Das A et al); and others
Review of Objectives

• at end of session, participants should be 

able to...

– consider sociotechnological approach to 

implementing IT in VA health care settings

– identify several key stakeholders
Working with Stakeholders

PCPs

IRMS

Clinical Applications 

Coordinators

Admin/

Clinical Mgrs

programming

networking

Clinic computer

Support staff

PAD LD

SJ and

VAMC

Stckn

Mod

ACOS Amb Care

Dep COS for LD

Med Serv Chief