# Presentation: 422158

## Research in Human-Computer Interaction for VA Clinical Decision Support

- July 15th, 2008
- Jason J. Saleem, PhD
- VA HSR&D Center on Implementing Evidence-Based Practice;
- IU Center for Health Services & Outcomes Research, Regenstrief Institute

## Presentation Overview

- Brief overview of human factors and HCI
- Human factors studies involving VA CPRS and computerized clinical reminders
- Current work: AHRQ grant (Doebbeling, PI); “Improving Integration of CDS into Workflow”
- Human-Computer Interaction (HCI) / IT Lab
- Questions / Discussion

## Human Factors Engineering

- Study of human physical and cognitive capabilities and limitations, and application of that knowledge to system design
- Design of interfaces between people and technology
- Human-machine interface technology
- Human-environment interface technology
- Human-job interface technology
- Human-software interface technology
*Human-computer interaction*
- Human-organization interface technology

## Common Human Factors Methods

- Usability study
- Performance-based (time on task, error rates)
- Scenario-based (think aloud technique)
- Simulation study
- Heuristic evaluation
- Cognitive task analysis
- Card sorting
- Ethnography / naturalistic observation
- Kushniruk and Patel – methods review in Journal of Biomedical Informatics - 2004

## Human Factors Studies – VA Computerized Clinical Reminders

- Observational Field Study
- Barriers and facilitations to clinical reminder use
*Saleem JJ, Patterson ES, Militello L, Render ML, Orshansky G, Asch SM.  Exploring barriers and facilitators to the use of computerized clinical reminders.  J Am Med Inform Assoc. 2005;12(4):438-47.*
- Follow-up Simulation Study
- A vs B comparison study of redesign recommendations
*Saleem JJ, Patterson ES, Militello L, et al.  Impact of clinical reminder redesign on learnability, efficiency, usability, and workload for ambulatory clinic nurses.  J Am Med Inform Assoc. 2007;14(5):632-40.*

- _This work funded by_:
- VA HSR&D Merit Review grant (TRX 02-216):
- “Human Factors and the Effectiveness of Computerized Clinical Reminders”
- Principal Investigators: Emily S. Patterson, PhD and Steven M. Asch, MD, MPH 7/03 – 6/06

## Observational Field Study

- Ethnographic, “naturalistic” observation
- Non-intrusive; shadowing of nurses and physicians
- 3 observers, 4 VA hospitals, 2 days/site
- Capture observable activities and verbalizations
- Self-report data about how artifacts (tools) support or hinder performance
- Qualitative field data

## Field Study Participants

## Field Study Results

- Barrier 1: Coordination Between Nurses and Providers
- Barrier 2: Satisfying Reminders While Not With the Patient
- Barrier 3: Workload
- Barrier 4: Lack of Flexibility
- Barrier 5: Poor Usability

## Facilitators

- Limit the number of reminders
- Position computer workstations strategically
- Improve integration of reminders into clinical workflow
- Feedback mechanism
- .

## Simulation Study

- A vs B comparison study
- 16 non-VA nurses
- Hypotheses: A redesigned interface compared to the current design will:
- Have greater learnability
- Have increased efficiency
- Have a lower perceived workload
- Have greater perceived user satisfaction

- Saleem JJ, Patterson ES, Militello L, et al. Impact of clinical reminder redesign on learnability, efficiency, usability, and workload for ambulatory clinic nurses. J Am Med Inform Assoc. 2007;14(5):632-640.

## *Fictitious patient record*

***Fictitious patient record***

## Slide 12

## Design changes

***Fictitious patient record***

****Prototype****

## Dependent Measures

| Attribute | Measure or Tool |
| --- | --- |
| Learnability | Time to reach proficiency |
| Efficiency | Time to complete scenarios |
| Perceived workload | NASA Task Load Index (TLX) |
| User satisfaction | Usability questionnaire |

## NASA Task Load Index (TLX)

- NASA Task Load Index (TLX)

## Results - Learnability

- Time limit = 300 sec (5 min)
- Learnability is statistically significant; time to satisfy a clinical reminder with design B (redesign) significantly less than time with design A (current system)
- t-test
- t stat = 4.365
- t.05, 14 = 1.761
- p < 0.001

- A = current CR design
- B = redesigned prototype

## Results - Efficiency

- Paired t-test revealed users completed 2 of 5 patient scenarios with significantly less time for Design B (p<0.05).

## Results – Workload: Mental Demand Subscale

- Mental demand with design B rated significantly less than with design A
- Means: A = 39.6, B = 34.2
- Paired t test
- t stat = 1.840
- t.05, 15 = 1.753
- p = 0.043

## Results – Workload: Frustration Subscale

- Frustration with design B rated significantly less than with design A
- Means: A = 33.8, B = 27.7
- Paired t test
- t stat = 2.040
- t.05, 15 = 1.753
- p = 0.030

## Usability Questionnaire

- Likert-type scale (1=strongly disagree; 7=strongly agree)
- Sample questions:
- The organization of the information on the systems screens is clear.
- The display layouts simplify tasks.
- The sequence of displays is confusing.

*See Saleem et al. JAMIA. 2007;14(5):632-640 for complete results*

## Implications for Design

- Modest design modifications to better integrate clinical reminders with CPRS increased time to reach proficiency in reminder use
- This could potentially impact the willingness for new users to adopt and consistently use clinical reminders

## Conclusions

- Human factors methods should be routinely used to rapidly collect data to support design decisions formatively (i.e., prior to implementation)
- improve user performance and usability
- reduce cost by addressing design issues pre-implementation
- This model is still not widely adopted in healthcare

## AHRQ Grant – Improving Integration of CDS into Workflow (Doebbeling, PI)

- AHRQ ACTION Collaborative
- Improving Quality Through Health IT (RFTO#8)
- CDS for Colorectal Cancer Screening
- Team:
- Brad Doebbeling, MD, MSc
- David Haggstrom, MD, MAS
- Jason Saleem, PhD
- Laura Militello, MA
- Heather Hagg, MS
- Jeff Linder, MD, MPH
- Paul Dexter, MD
- Brian Dixon, MPA
- et al.

## Study Objectives

- Identify key approaches for effective integration of CDS for colorectal cancer screening into clinical workflow.
- Test alternatives through controlled simulation analysis.
- Evaluate improved CDS design after subsequent implementation (at a VA test-site).

## Slide 25

## Phase 1 - Observational Sites

- VAMC – West Haven, CT
- CRC screening computerized clinical reminder
- VAMC – Columbia, SC
- OncWatch CDS (clinical reminder and management tool)
- Divides patients (based on the data) into four different cohorts, defined by their risk/needs
- Creates a “fail safe” system to identify patients and ensure follow-up recommendations are being fulfilled
- Regenstrief Institute – 2 Indianapolis clinics
- Encounter form reminders for CRC screening
- Partners Healthcare – Brigham & Women’s Hospital
- Previous failed attempts to implement CRC screening CDS: “no easy way to feed back that an adequate colonoscopy was done and was normal”
- New CRC screening tool being piloted this year
- Phases 2 & 3 – 2009

## Summary

- Clinical reminder studies and current work with AHRQ on CDS workflow integration:
- Qualitative field observation followed by scenario-driven, comparative usability testing of experimental prototypes in a simulated setting
- Clinical software development may benefit from this paradigm if more widely followed
- Acknowledgments:
- Emily S. Patterson, PhD
- Steven M. Asch, MD, MPH
- Marta L. Render, MD
- Bradley N. Doebbeling, MD, MSc

## Human-Computer Interaction (HCI) / IT Lab (Indianapolis VAMC)

- Operational – May 2008
- Rapid prototyping of design alternatives
- Usability testing / simulation study
- Specialized software and data collection equipment
- VA network & University network
- Grad students specializing in usability and human-computer interaction

## Perfect Timing?

- Just published:
- Sittig DF, Wright A, Osheroff JA, Middleton B, Teich JM, Ash JS, Campbell E, Bates DW. **“Grand Challenges in Clinical Decision Support v11”**, Journal of Biomedical Informatics, 2008;41(2):387-92.
- #1 Ranked challenge in clinical decision support: **“Improve the human-computer interface”**.
- Paper was written to influence funders and policy makers

## Human-Computer Interaction

**Human-Computer Interaction**
**Information Technology**
**Laboratory**
**(HCI / IT LAB)**
**D-5014**

_**VA Laptop**_
- Morae
- Snag-it

_**VA Desktops**_
- Visual Basic

_**IU Laptop**_
- Morae
- Snag-it

_**IU Desktops**_
- Visual Basic
- Azure X 1PC

**- VA LAN OUTLET**

**- IU LAN OUTLET**

## Current Project Portfolio for HCI / IT Lab; affiliated projects

- _Funded_:
- VA STP: “Factors Influencing Effective Implementation of My HealtheVet” (Chumbler, PI)
- AHRQ RFTO#8 HIT: “Implementing and Improving the Integration of Decision Support into Outpatient Clinical Workflow” (Doebbeling, PI)
- VA CDA: “Colorectal Cancer Survivor Surveillance Care and Personal Health Records” (Haggstrom)
- Purdue– Lilly Seed Grant: “Integrating Pharmacogenomic-guided Dosing into Clinical Practice” (Overholser, PI)
- Doctoral Dissertation: “Impact of Information Flow and Prioritization on the Use of Computerized Clinical Reminders” (Wu)
- _In Process_:
- VA IIR for June ‘08: “The Effects of Exam Room Computing on Patient-Centered Communication” (Frankel, PI)
- VA IIR for Dec ‘08: “Barriers and Facilitators to Providers’ Adoption of My HealtheVet” (Chumbler, PI)
- VA CDA or IIR for Dec ‘08: “Circumventing Health IT with Paper: Identifying Patient Safety Risks” (Saleem)

## Focus Areas

- Rapid prototyping of CDS and other Health IT
- Applied usability studies
- Research experiments through simulation (as part of a broader research agenda, including field study)
- Knowledge Management capabilities
- Prototype KM strategies

## Goals

- Become a warehouse for user-centered design
- Provide early input through rigorous human factors methods for health IT software and hardware design
- Become a resource for healthcare informatics research
- Design research studies to improve health IT prior to implementation
- Become a bridge for collaboration VHA-wide and external to VA
- Partnerships with RI, IU School of Informatics, Purdue, etc.
- Partnership with Roudebush VA and VISN 11 CIO on local projects;
- Partnership with VA Office of Information & Technology on national-level projects
- Partnership with non-VA research groups on broad health IT projects (e.g., CDS Consortium with Partners Healthcare, Regenstrief, etc.)
- Actively seek strategic teaming opportunities for collaborative research to transfer knowledge out of the Lab to meet real world challenges

## Opportunities / Partnerships

- Resource for Center Investigators to conduct studies as part of an overall research proposal or CDA
- Pilot studies for rapid data collection on usability and piloting of new technology to support future proposals
- Demonstration of Center’s IT research capabilities to visitors / collaborators
- Graduate students, internships to support work in the Lab
- Potential partnership with VA Office of Information & Technology to provide input during clinical applications development, usability testing and assess impact on patients and providers
- Potential partnership with Roudebush VA and VISN 11 CIO on local projects

## Contact

- Jason J. Saleem, PhD
- Human Factors Engineer
- VA Center on Implementing Evidence-Based Practice
- IU Center for Health Services & Outcomes Research, Regenstrief Institute
- _[jsaleem@iupui.edu](mailto:jsaleem@iupui.edu)_