# Presentation: 687538

## MHV Usability Testing

- Neale R. Chumbler, PhD, Jason Saleem, PhD, David Haggstrom, MD, MAS
- VA HSR&D CIEBP and Stroke QUERI

## Acknowledgements

- Alissa Russ, PhD, Josette Jones, PhD, Scott Russell, BS, Wanda Hines
- The research reported/outlined here was supported by the Department of Veterans Affairs, Veterans Health Administration, Health Services Research and Development Service (SHP 08-192). Dr. Chumbler is the Co-PI of the HSR&D CIEBP at Roudebush VAMC.

## Personal Health Records

- Holds great promise in promoting patient-centered care
- Very few studies have tested the efficacy or implementation of PHR systems.
- WSJ/Harris---75% of Americans would communicate with their physicians if given means to do so
- 2nd survey --- 60% of patients would look up test results & track medication use if records were available.

- Kaelber et al., 2008

## Personal Health Records

- VA’s PHR, MHV, has reported satisfaction survey results.
- Many positive findings
- Room for improvement in:
- Ease of navigation
- Need for more usability testing

## AIMS

- Identify factors impacting patient use of MHV
- Explore how to engage patients in the best use and navigation of MHV
- Usability testing is congruent with the MHV Evaluation framework foci

## MHV Evaluation Framework

- MHV framework has adopted the RE-AIM Performance Framework as a tool to organize and synthesize work on MHV.
- Our work is congruent:
- Functionality
- Navigation
- Search

## What is Usability

- Usability measures the quality of a user’s experience when interacting with a product or system
- Its not a single one-dimensional property. It has been associated with five attributes:
- Learnability		Low error rate
- Efficiency		Satisfaction
- Memorability

## What is Usability Testing

- Usability testing is carrying out experiments to find out specific information about a design.
- Evaluating a single design
- Comparing two or more designs
- Determining if a design works in real life
- 		(ideally done *prior* to full implementation)
- Meant for rapid feedback to design team so changes can be made iteratively throughout the design cycle.

## Factors Influencing Implementation of MHV

- Phase I --- semi-structured interviews with 8 patients who previously used MHV
- How often do you use MHV?
- Are their specific things that prevent you form using MHV more frequently?
- What type of things do you use MHV for?
- Have you experienced frustration in finding information in MHV?
- Are there specific functions in MHV that you find difficult to use?

## Usability Test of MHV

- HCI/IT lab...n = 24...simulate a typical setting that a patient would access MHV.
- Qualitative interviews helped inform the design of the usability scenarios
- For example...
- Prescription refill (10 minutes)

## Usability Metrics

- Quantitative ---
- 1) Efficiency (time to complete scenarios)
- 		through time stamped Morae digitial video recordings
- 	2) Error rates (observed deviations from intended use of MHV)
- 	3) Satisfaction (7-point Likert Scale)

## Initial Usability Results (In Progress)

- Rx refill time on task
- No significant difference in actual time task performance (mean = 209 sec) for 1 Rx refill and target task performance (240 sec; 4 min) for first Rx refill task
- Actual task performance (mean = 102 sec) for same Rx refill task after second run was significantly lower than target task performance (240 sec; 4 min); p=.007

## Initial Usability Results (In Progress)

- Participants rated two usability statements related to error messages and recovery from mistakes significantly lower than target usability levels
- (target = 5 on a 7 point Likert-type scale where 1 = strongly disagree and 7 = strongly agree)
*The system gives error messages that clearly tell me how to fix problems. *(mean rating = 3.2), p-value = .001
*Whenever I make a mistake using the system, I recover easily and quickly. *(mean rating = 3.5), p-value = .001

## Lessons Learned

- Usability testing can provide important and prompt design feedback to increase user satisfaction through participation of actual end users.
- Collaboration with central office partners (Kim Nazi) increases value of work to VA.
-  A priori usability targets, like a priori hypotheses, are of greatest value from study design standpoint.
- Basic steps in application design should not be overlooked
- e.g., log-in functions

## Plans for Moving Ahead

- Consider usability scenarios for disease-specific conditions
- e.g., cancer, stroke, diabetes
- Further integrate MHV usability research agenda with work of MHV design team