Research in Human-Computer Interaction for VA Clinical Decision Support

July 15th, 2008

Jason J. Saleem, PhD

VA HSR&D Center on Implementing Evidence-Based Practice;

IU Center for Health Services & Outcomes Research, Regenstrief Institute

*

Presentation Overview
        - Brief overview of human factors and HCI
        - Human factors studies involving VA CPRS and computerized clinical reminders
        - Current work: AHRQ grant (Doebbeling, PI); “Improving Integration of CDS into Workflow”
        - Human-Computer Interaction (HCI) / IT Lab
        - Questions / Discussion

*

Human Factors Engineering
        - Study of human physical and cognitive capabilities and limitations, and application of that knowledge to system design
        - Design of interfaces between people and technology
        - Human-machine interface technology
        - Human-environment interface technology
        - Human-job interface technology
        - Human-software interface technology
        - Human-computer interaction
        - Human-organization interface technology

*

Common Human Factors Methods
        - Usability study
        - Performance-based (time on task, error rates)
        - Scenario-based (think aloud technique)
        - Simulation study
        - Heuristic evaluation
        - Cognitive task analysis
        - Card sorting
        - Ethnography / naturalistic observation
        - Kushniruk and Patel – methods review in Journal of Biomedical Informatics - 2004

Human Factors Studies – VA Computerized Clinical Reminders

*

Observational Field Study
        - Barriers and facilitations to clinical reminder use

Saleem JJ, Patterson ES, Militello L, Render ML, Orshansky G, Asch SM.  Exploring barriers and facilitators to the use of computerized clinical reminders.  J Am Med Inform Assoc. 2005;12(4):438-47.

Follow-up Simulation Study
        - A vs B comparison study of redesign recommendations

Saleem JJ, Patterson ES, Militello L, et al.  Impact of clinical reminder redesign on learnability, efficiency, usability, and workload for ambulatory clinic nurses.  J Am Med Inform Assoc. 2007;14(5):632-40.

This work funded by:

VA HSR&D Merit Review grant (TRX 02-216): 

“Human Factors and the Effectiveness of Computerized Clinical Reminders”

Principal Investigators: Emily S. Patterson, PhD and Steven M. Asch, MD, MPH 7/03 – 6/06

Jason J. Saleem, PhD

*

Observational Field Study
        - Ethnographic, “naturalistic” observation
        - Non-intrusive; shadowing of nurses and physicians
        - 3 observers, 4 VA hospitals, 2 days/site
        - Capture observable activities and verbalizations
        - Self-report data about how artifacts (tools) support or hinder performance
        - Qualitative field data

Jason J. Saleem, PhD

Jason J. Saleem, PhD

*

Field Study Participants

Jason J. Saleem, PhD

Jason J. Saleem, PhD

*

Field Study Results
        - Barrier 1: Coordination Between Nurses and Providers
        - Barrier 2: Satisfying Reminders While Not With the Patient
        - Barrier 3: Workload
        - Barrier 4: Lack of Flexibility
        - Barrier 5: Poor Usability

Jason J. Saleem, PhD

Jason J. Saleem, PhD

*

Facilitators
        - Limit the number of reminders
        - Position computer workstations strategically
        - Improve integration of reminders into clinical workflow
        - Feedback mechanism

.

Jason J. Saleem, PhD

*

Simulation Study
        - A vs B comparison study
        - 16 non-VA nurses
        - Hypotheses: A redesigned interface compared to the current design will:

Have greater learnability

Have increased efficiency

Have a lower perceived workload 

Have greater perceived user satisfaction

Saleem JJ, Patterson ES, Militello L, et al. Impact of clinical reminder redesign on learnability, efficiency, usability, and workload for ambulatory clinic nurses. J Am Med Inform Assoc. 2007;14(5):632-640.

*

*Fictitious patient record*

*

        - Design changes

*

*Fictitious patient record*

*Prototype*

*

Dependent Measures
| Attribute          | Measure or Tool            |
|--------------------|----------------------------|
| Learnability       | Time to reach proficiency  |
| Efficiency         | Time to complete scenarios |
| Perceived workload | NASA Task Load Index (TLX) |
| User satisfaction  | Usability questionnaire    |

*

NASA Task Load Index (TLX)

*

Results - Learnability
        - Time limit = 300 sec (5 min)
        - Learnability is statistically significant; time to satisfy a clinical reminder with design B (redesign) significantly less than time with design A (current system)
        - t-test
        - t stat = 4.365
        - t.05, 14 = 1.761
        - p < 0.001

A = current CR design

B = redesigned prototype

*

Results - Efficiency
        - Paired t-test revealed users completed 2 of 5 patient scenarios with significantly less time for Design B (p<0.05).

*

Results – Workload: Mental Demand Subscale
        - Mental demand with design B rated significantly less than with design A
        - Means: A = 39.6, B = 34.2
        - Paired t test
        - t stat = 1.840
        - t.05, 15 = 1.753
        - p = 0.043

*

Results – Workload: Frustration Subscale
        - Frustration with design B rated significantly less than with design A
        - Means: A = 33.8, B = 27.7
        - Paired t test
        - t stat = 2.040
        - t.05, 15 = 1.753
        - p = 0.030

*

Usability Questionnaire
        - Likert-type scale (1=strongly disagree; 7=strongly agree)
        - Sample questions:
        - The organization of the information on the systems screens is clear. 
        - The display layouts simplify tasks.
        - The sequence of displays is confusing.

See Saleem et al. JAMIA. 2007;14(5):632-640 for complete results

*

Implications for Design
        - Modest design modifications to better integrate clinical reminders with CPRS increased time to reach proficiency in reminder use
        - This could potentially impact the willingness for new users to adopt and consistently use clinical reminders

*

Conclusions
        - Human factors methods should be routinely used to rapidly collect data to support design decisions formatively (i.e., prior to implementation)
        - improve user performance and usability
        - reduce cost by addressing design issues pre-implementation
        - This model is still not widely adopted in healthcare

*

AHRQ Grant – Improving Integration of CDS into Workflow (Doebbeling, PI)
        - AHRQ ACTION Collaborative
        - Improving Quality Through Health IT (RFTO#8)
        - CDS for Colorectal Cancer Screening
        - Team: 
        - Brad Doebbeling, MD, MSc
        - David Haggstrom, MD, MAS
        - Jason Saleem, PhD 
        - Laura Militello, MA 
        - Heather Hagg, MS
        - Jeff Linder, MD, MPH
        - Paul Dexter, MD
        - Brian Dixon, MPA
        - et al.

*

Study Objectives

Identify key approaches for effective integration of CDS for colorectal cancer screening into clinical workflow. 

Test alternatives through controlled simulation analysis. 

Evaluate improved CDS design after subsequent implementation (at a VA test-site).

*

Key Informant Interviews of site-specific best-practices for integration of colorectal cancer screening CDS into workflow

Direct Observation of colorectal cancer screening CDS for barriers and facilitators to workflow integration

Implementation

in primary care clinic after simulation study

Rapid Prototyping of CDS design alternatives based on Phase 1 findings

Simulation Study to test impact of CDS design alternatives on efficiency, usability, and workload

Evaluation in primary care clinic after simulation study

Figure.  Project Overview

Phase 1

Phase 2

Phase 3

*

Phase 1 - Observational Sites
        - VAMC – West Haven, CT
        - CRC screening computerized clinical reminder
        - VAMC – Columbia, SC
        - OncWatch CDS (clinical reminder and management tool)
        - Divides patients (based on the data) into four different cohorts, defined by their risk/needs
        - Creates a “fail safe” system to identify patients and ensure follow-up recommendations are being fulfilled
        - Regenstrief Institute – 2 Indianapolis clinics
        - Encounter form reminders for CRC screening
        - Partners Healthcare – Brigham & Women’s Hospital
        - Previous failed attempts to implement CRC screening CDS: “no easy way to feed back that an adequate colonoscopy was done and was normal”
        - New CRC screening tool being piloted this year
        - Phases 2 & 3 – 2009

Summary
        - Clinical reminder studies and current work with AHRQ on CDS workflow integration:
        - Qualitative field observation followed by scenario-driven, comparative usability testing of experimental prototypes in a simulated setting
        - Clinical software development may benefit from this paradigm if more widely followed
        - Acknowledgments:
        - Emily S. Patterson, PhD
        - Steven M. Asch, MD, MPH
        - Marta L. Render, MD
        - Bradley N. Doebbeling, MD, MSc

*

*

Human-Computer Interaction (HCI) / IT Lab (Indianapolis VAMC)
        - Operational – May 2008
        - Rapid prototyping of design alternatives
        - Usability testing / simulation study
        - Specialized software and data collection equipment
        - VA network & University network
        - Grad students specializing in usability and human-computer interaction

*

Perfect Timing?

Just published:
        - Sittig DF, Wright A, Osheroff JA, Middleton B, Teich JM, Ash JS, Campbell E, Bates DW. “Grand Challenges in Clinical Decision Support v11”, Journal of Biomedical Informatics, 2008;41(2):387-92.
        - #1 Ranked challenge in clinical decision support: “Improve the human-computer interface”.
        - Paper was written to influence funders and policy makers

*

Human-Computer Interaction

Information Technology

Laboratory

(HCI / IT LAB)

D-5014

VA Laptop
        - Morae
        - Snag-it

VA Desktops
        - Visual Basic

IU Laptop
        - Morae
        - Snag-it

IU Desktops
        - Visual Basic
        - Azure X 1PC

- VA LAN OUTLET

- IU LAN OUTLET

47 inch monitor

17” monitor & player for

researcher’s office

Portable 

DigiCam

w/WA  adapter

*

Current Project Portfolio for HCI / IT Lab; affiliated projects

Funded:
        - VA STP: “Factors Influencing Effective Implementation of My HealtheVet” (Chumbler, PI) 
        - AHRQ RFTO#8 HIT: “Implementing and Improving the Integration of Decision Support into Outpatient Clinical Workflow” (Doebbeling, PI)
        - VA CDA: “Colorectal Cancer Survivor Surveillance Care and Personal Health Records” (Haggstrom)
        - Purdue– Lilly Seed Grant: “Integrating Pharmacogenomic-guided Dosing into Clinical Practice” (Overholser, PI) 
        - Doctoral Dissertation: “Impact of Information Flow and Prioritization on the Use of Computerized Clinical Reminders” (Wu)

In Process:
        - VA IIR for June ‘08: “The Effects of Exam Room Computing on Patient-Centered Communication” (Frankel, PI)
        - VA IIR for Dec ‘08: “Barriers and Facilitators to Providers’ Adoption of My HealtheVet” (Chumbler, PI)
        - VA CDA or IIR for Dec ‘08: “Circumventing Health IT with Paper: Identifying Patient Safety Risks” (Saleem)

*

Focus Areas
        - Rapid prototyping of CDS and other Health IT
        - Applied usability studies
        - Research experiments through simulation (as part of a broader research agenda, including field study)
        - Knowledge Management capabilities
        - Prototype KM strategies

*

Goals

Become a warehouse for user-centered design
        - Provide early input through rigorous human factors methods for health IT software and hardware design

Become a resource for healthcare informatics research
        - Design research studies to improve health IT prior to implementation

Become a bridge for collaboration VHA-wide and external to VA
        - Partnerships with RI, IU School of Informatics, Purdue, etc.
        - Partnership with Roudebush VA and VISN 11 CIO on local projects; 
        - Partnership with VA Office of Information & Technology on national-level projects
        - Partnership with non-VA research groups on broad health IT projects (e.g., CDS Consortium with Partners Healthcare, Regenstrief, etc.)
        - Actively seek strategic teaming opportunities for collaborative research to transfer knowledge out of the Lab to meet real world challenges

*

Opportunities / Partnerships
        - Resource for Center Investigators to conduct studies as part of an overall research proposal or CDA
        - Pilot studies for rapid data collection on usability and piloting of new technology to support future proposals
        - Demonstration of Center’s IT research capabilities to visitors / collaborators
        - Graduate students, internships to support work in the Lab
        - Potential partnership with VA Office of Information & Technology to provide input during clinical applications development, usability testing and assess impact on patients and providers
        - Potential partnership with Roudebush VA and VISN 11 CIO on local projects

*

Contact

Jason J. Saleem, PhD

Human Factors Engineer

VA Center on Implementing Evidence-Based Practice

IU Center for Health Services & Outcomes Research, Regenstrief Institute

jsaleem@iupui.edu