#### Human Factors Engineering

#### Usability Testing Exercise

By John Gosbee, MD

VA National Center for Patient Safety

<John.Gosbee@med.va.gov> 734-930-5893 www.patientsafety.gov

##### Background and Rationale

Most introductory courses in human factors engineering include hands-on
exercises. The "systems" mind-set (turning people's brains around 180
degrees) is rarely developed with didactic approaches. Instead, doing
usability testing to uncover design issues has been found to create
"teachable moments" (Sojourner, et al, 1993). This is mainly true
because it is hard to teach the concept of "learned intuition", which
is: 1) you cannot recall ever not knowing how to use something; and 2)
cannot imagine someone else would not know how it "should" be used.

In addition, usability testing projects have been used with success for
medical and PharmD students, and residents in month-long rotations, as
well as nursing students in the classroom (Gosbee, 1998; Stahlhut, et al
1997).

##### Objectives

1\) Create teachable moment for human factors concepts and framework

2\) Create a lasting impression, long after they forgot your words

3\) Demonstrate common design flaws found in medical devices, software,
and work areas

##### Applicability

Almost anyone at any level could benefit. Some version of this exercise
has been tried successfully on 16 year-old high school kids in Junior
Achievement all the way to upper management at medical device companies.
Typically there is 60-80% "aha" rate.

##### Constraints and Lessons Learned

Almost no constraints have been found. Need at least 30-40 minutes, but
60-90 minutes is better. Try summarizing findings and suggested
redesigns on a flip chart or white board. Consider organizing those
findings across human factors engineering principles. You can't be too
obvious in drawing analogies to medical setting and systems.

####### Description of Typical Usability Testing

In formal usability testing, many end-users attempt to use the system
while following likely scenarios under conditions that mimic actual
operation. Measurements can be made of errors, recovery from errors, and
time to complete a task. Some evaluations include "softer" measures like
personal opinion or reflection. The key is to uncover mental models,
concepts, and hidden vulnerabilities within the system. The activity is
a "biopsy" of sorts to find adverse events waiting to happen. It is not,
however, to be confused with an expert evaluation or "focus group"

##### Procedure (Outline) for Exercise

###### Short List of Systems to Buy and Use

Travel packs of baby wipes (or other travel packs, like hand cleaner,
lens cleaner)

Mint dispensers (e.g., SMINTS)

Soft foil packs of pink cool-ade (e.g., MinuteMaid)

Travel packs of Breath-Rite strips

Calculators (almost any type, but beware of massive irritation)

Paper "screens" of "simple" medical software interfaces (need more time
& instruction)

###### Doing the Usability Evaluation

-   Groups of 3-4 people, sitting around a common table or work surface

> **Director**
>
> a\) Reminds end-user to think aloud
>
> b\) Leads team discussion about issues and redesign
> [after]{.underline} usability test is complete
>
> **End-User**  uses the system as they would (without interference or
> help)
>
> **Observers**
>
> a\) Document end-users actions and words
>
> b\) Also facial expressions, swearing, etc.

-   Once the person has completed the task of using the system (without
    guidance), the team

a\) **Summarizes issues** related to observed behavior, comments, etc.

b\) **Recommends redesign** related to issues

###### Discussion and Report

> 1\) Each team reports one issue and redesign (until each team has one
> turn)
>
> 2\) Discussion is led towards common themes for design deficiencies
> (human factors engineering flaws)
>
> a\) Consistency of labels, buttons, widgets
>
> b\) Readable and understandable labels
>
> c\) Obvious and understandable model
>
> d\) "Getting lost" (situational awareness)
>
> e\) What is it doing and why is it doing that?
>
> f\) "Mode" errors
>
> g\) Negative transfer of training
>
> 3\) The discussion also explicitly ties the issues and redesigns to
> medical devices, consumables, or software examples (e.g., poor
> labeling might occur on medication packets for patients)
>
> 4\) Try summarizing findings and suggested redesigns on a flip chart
> or white board.

####### Advisories

The instructions usually have to be explained a few times and rationale
provided. If you don't, many teams will simply handle and talk about the
items. If they do this, they find only [superficial issues]{.underline}
and are only able to tie their ideas to "possible" errors. Consider not
giving the item to each team unless they have picked the End-User and
Director, and the observers have paper and pen ready to record the very
first thing the End-User does when they are handed the object.

**Further Discussion and Examples** (if time allows)

1\) Secretaries using mouses for the first time in the early 1980s --
thought they were foot pedals like dictation machines or sewing machines
and put them on the floor.

2\) Secretaries using floppy disks for the first time in the early 1980s
-- thought they were vinyl records and opened them "all the way up".

3\) IV Pump had "program \# two" screen that looked like "program \#
one" screen

-   Pump returns to home care operations center and is supposed to have
    Program #1 cleared

-   It is not cleared and is cleaned and sent to be reprogrammed by the
    pharmacist

-   Pharmacist turns pump on to program it, but it defaults to Program
    #2

-   There is a small, easily missed icon denoting the program mode

-   Pump is given to the home care nurse, who turns it on and it
    defaults to Program #1

4\) Plug for pediatric EKG leads that looked like it should be plugged
into an electrical socket, and was...

5\) Labels for medication vial have 5mg/ml in large bold type, with 5ml
in small type at bottom. The nurse was asked to draw up and give 5mg,
and gave the whole vial...

**[References]{.underline}**

Bridger RS, Poluta MA. Ergonomics: Introducing the human factor into the
clinical setting. *J of Clinical Engineering* 1998; 23: 180-188.

Gosbee, J.W., Arnecke, B., Klancher, J., Wurster, H., Scanlon, M.
(2001). The Role of Usability Testing in Healthcare Organizations. In
*Proceedings of the Human Factors Society 40th Annual Meeting.* Santa
Monica, CA: Human Factors Society, pp. 1308-1311.

##### This summary is included in the following pages of this document 

Nielsen J. *Usability Engineering*. Cambridge, MA: AP Professional,
1993.

Saladow J. Continuum of care and human factors design issues. *J
Intraven Nurs* 1996; 19(3 Suppl): 20-24.

Shapiro RG, Fox JE. Games to explain human factors. In *Proceedings of
the Human Factors and Ergonomics Society 45^th^ Annual Meeting.* Santa
Monica, CA: Human Factors and Ergonomics Society, 2001.

Sojourner RJ, Aretz AJ,Vance KM. Teaching an Introductory Course in
Human Factors Engineering: A Successful Learning Experience. In
*Proceedings of the Human Factors and Ergonomics Society 33^rd^ Annual
Meeting.* Santa Monica, CA: Human Factors and Ergonomics Society, 1993

Stanhope N, Vincent CA, Adams S, O\'Connor AM, Beard RW. Applying human
factors methods to clinical risk management in obstetrics. *Br J Obstet
Gynaecol* 1997; 104: 1225-1232.

Wiklund ME. *Medical device and equipment design: Usability engineering
and ergonomics.* Buffalo Grove, IL: Interpharm Press, 1995.

####  

#### The Role of Usability Testing in Healthcare Organizations

##### 

##### *Human Factors and Ergonomics Society 45^th^ Annual Meeting, October 8-12, 2001 (pp. 1308-1311)*

##### John Gosbee; Dept. of Veterans Affairs National Center for Patient Safety; Ann Arbor, MI

Joe Klancher; Allina Health System; Minneapolis, MN

##### Bernadette Arnecke; Desert Samaritan Medical Center; Banner Health System; Mesa, AZ

Heather Wurster; University of Michigan Hospitals and Clinics; Ann
Arbor, MI

Matthew Scanlon; Medical College of Wisconsin; Milwaukee, WI

##### This panel included practical examples of how physicians, nurses, and quality managers have utilized usability testing activities to help their healthcare organization. Some used it to provide crucial information to support procurement decisions. Others used data from usability testing to understand where to focus training and other implementation efforts. Usability testing helped some create teachable moments while imparting the concepts and framework of human factors engineering. Some panelists will talk about the aspects of healthcare organizational culture that stand in the way of implementing usability testing. All panelists will discuss the key decision makers in their health care organization who should be part of this important effort. 

######### 

######### BACKGROUND AND RATIONALE

As in many domains, the lack of human factors engineering process can
lead to some very unfortunate consequences. A plug for pediatric EKG
leads looked like it should have been plugged into an electrical socket,
and was. Labels for medication vial have 5mg/ml in large bold type, with
5ml as total volume in small type at bottom of the vial. The nurse was
asked to draw up and give 5mg, and gave the whole vial. Instructions for
Defibrillator leads on automated external defibrillator (AED) were
located on the peel-away paper covers...then the wind blew them away
while policeman were using the AED outdoors.

There is much advice about how medical device companies should embrace
human factors engineering (Wiklund, 1995). However, it has been
recommended that human factors engineering should play a key role in
health care delivery organizations, such as hospitals (Welch 1998). It
is not entirely clear how such techniques like usability testing would
actually be used. This panel will contain practical information from
healthcare providers about the role of usability testing in changing
decision-making, policy, and the mindset of key stakeholders.

To incorporate human factors engineering into any organization, one of
the first steps is awareness and training. In healthcare, there has been
huge increase in awareness of errors and the role human factors
engineering could play (Kohn, Corrigan, & Donaldson, 1999). There is
much less understanding and agreement on the best way to educate and
implement human factors engineering principles and tools.

The human factors engineering mind-set (turning people's brains around
180 degrees) is rarely accomplished with didactic approaches. This is
mainly true because it is hard to teach the concept of "learned
intuition". Briefly, learned intuition is the phenomena where you cannot
recall ever not knowing how to use something; and you cannot imagine
someone else would not know how it "should" be used. Educators have
created "teachable moments" about learned intuition by having students
grapple with human factors design problems and having them develop and
test remedies (Sojourner, et al, 1993). Also, incorporating usability
testing into resident, medical, pharmacy, and nursing student training
has been used with success (Gosbee, 1998).

There has been some literature on successful incorporation of human
factors engineering and usability testing into health care
organizations. Welch (1998) proposed avoiding troublesome devices and
software by both performing usability testing of products before
purchase and demanding usability testing data from manufacturers. The
Mayo Clinic in Rochester, Minnesota built usability (i.e., human
factors) labs to evaluate many software products before purchase -- as
well as use them for their own development projects (Claus, et al.
1997). Some in the patient safety improvement arena have cited the
importance of usability testing before purchase or during implementation
of a new product (Gosbee & Lin 2001). It is also recommended that
usability testing be integrated into "in-house" design and development
of software and complex work areas (Espinoza 2001). Finally, usability
testing might be an effective tool to train a human factors mindset for
clinicians and management Gosbee (1999).

In this panel, some presenters will provide practical examples of how
they have used usability testing activities to help their healthcare
organization, including: 1) providing crucial information to support
procurement decisions; 2) understanding where to focus training and
other implementation efforts; and 3) creating teachable moment for human
factors concepts and framework. Other panelists will talk about the
aspects of healthcare organizational culture that stand in the way of
implementing usability testing. All panelists will discuss the key
decision makers in their health care organization who should be part of
this important effort.

Some presenters will explain the challenges they faced -- and those
challenges that still remain. For example, diffuse responsibility for
the final decision on medical device purchases. Other presenters will
cite the perceived roadblocks that turned out not to be problems. For
example, many physicians and nurses were quickly excited about learning
human factors and usability testing -- and the excitement endured longer
than expected. All panelists will highlight the steps that led up to
"the light bulb going off over the heads" of key decision makers in
their particular situation.

##### Audience discussion will be led to focus on the following topics and others as suggested: 1) the best methods to replicate these successes in other settings; 2) the role that human factors academia could play; and 3) implications for human factors engineering curricula and internship experiences.

INDIVIDUAL PANEL SESSIONS

Teaching the Role of Usability Testing and Heuristic Evaluation to
Healthcare Providers and Management (John Gosbee)

After much interest in the practical aspects of human factors
engineering, Latiolais Leadership Program sponsored a 1.5 day workshop
on the role of human factors methods in the medication use process.
After 4 hours of didactic material and demonstrations, small group
exercises were used to impart the essentials of usability testing and
heuristic evaluation and help to engender a user-centered mindset. In
four workshops, groups of pharmacy managers, nurse managers, and device
company representatives talked about how to incorporate HFE methods into
their "real world". At the end of one of the workshops on heuristic
evaluation, 17 of 17 attendees said they conceptually agreed they should
adopt the method. A smaller subset, 7 or 17, stated they would attempt
to apply the learning as soon as they returned to their organization.
Reasons for hesitation included not enough training, infrastructure was
not in place, and the rationale and resources needed would be hard to
sell to management.

Determining the Role of Usability Testing for Pre-Purchase Decision
Making as Part of Patient Safety Improvement (Joe Klancher)

Allina Health System has been developing patient safety systems
including: workplace analysis, incident reporting, and accident
investigation systems, human error. Knowledge of human factors
engineering has been identified as one critical success factor in
developing and maintaining effective patient safety systems. Our
organization has been providing training to key staff in these areas and
working to implement usability testing in a useful manner.

The recent legislation mandating the evaluation and implementation of
safe needle devices provided an excellent opportunity to integrate human
factors analysis into our product selection process. As we implemented
our safe needle evaluation and selection process (which included
rigorous simulations and evaluation of products by end users) we
integrated human factors and other safety related criteria into the
evaluation process. Our "evaluation checklists" included consideration
of human factors engineering, as well.

These experiences provided an excellent learning opportunity for our
materials management experts and the frontline staff involved in product
selection. Currently work in being done to integrate the safety/human
factors evaluation process into medical equipment and commodities
selection.

##### Usability Testing in the Hospital Setting to Introduce New Products (Bernadette Arnecke)

Usability testing was introduced at Desert Samaritan Medical Center in
Mesa, Arizona, after our medical director learned the technique from an
Institute for Healthcare Improvement workshop (90 minutes). After our
organization made a decision to purchase a free-flow-protection IV pump,
we performed usability testing to anticipate training problems and
discover issues to communicate back to the vendor. Sessions included
nurses, clinical nurse specialists, a nurse educator, and a unit
manager. Observers included a manufacturer representative and the
hospital's product standardization and utilization coordinator. Typical
IV pump tasks were used with groups of three during hour-long sessions.
The experience was eye opening for all involved.

Other usability testing sessions included: 1) eight new employees with a
range of nursing experience performed usability testing on safety
hypodermic needles; and 2) training for material management
professionals, where the human factors didactic material was taught,
followed by usability testing training with foil pouch drinks and wipe
dispensers.

## 

Results so far include: 1) enhanced understanding of the user-device
interface by our nurse educators, which assisted training and
implementation of the new IV pumps; 2) change in approach by our product
standardization and utilization coordinator, who now tries to use
products without being taught by the vendors to see how intuitive and
usable the products are; 3) Expressed desire by our system materials
management director to incorporate usability testing into the product
selection process.

Nursing Culture and Practice may be Delaying Recognition and Adoption of
Human Factors Engineering and Usability Testing (Heather Wurster)

Many steps are required before a health care organization would use
human factors engineering and tools to help analyze new and existing
medical devices. Nurses are one key stakeholder group that uses and
occasionally helps select medical devices. In nursing practice and
management situations, this panelist has come to understand some ways
that nursing "culture" may be a roadblock. For one, some nurses are
valued if they can use devices without asking for help and not providing
negative feedback about a device. This often occurs due to time
constraints, complex treatment plans, and seemingly constant
introduction of new devices. In addition, "work-arounds" to meet patient
care goals are learned and applied early in a nurses training.
Therefore, it will be key to make nurses and other healthcare workers
aware of the downside to working through and around hard-to-use or
hard-to-learn medical systems. Several front-line stories will be used
to emphasize key points. Some lessons in how these norms might be
changed or addressed will be presented.

Issues when Applying Usability Testing to a Pediatric Teaching Hospital
(Matthew Scanlon)

Usability testing takes on added importance in the context of pediatric
care. Many devices used in the care of children were not designed for
pediatric use. Liability concerns have led manufacturers to limit design
to devices for adults leaving pediatric health care providers to apply
devices that may risk injury to children. For example, the absence of
feeding pumps for premature infants has led to the use of intravenous
medication pumps to deliver small volumes on formula. Patient injury and
death has occurred when care providers inadvertently infused formula,
administered with intravenous pumps through intravenous tubing, to an
intravenous catheter.

There are several challenges in applying usability testing in a
pediatric teaching hospital. One challenge is finding "teachable
moments" to apply adult learning principles to providers in a busy work
environment. Efforts include bedside discussions as well as "in-service"
sessions for new pumps. Another challenge is changing culture and
raising awareness of hospital employees as a physician not employed by
the hospital. This requires the education of hospital administrators
about the need for usability testing. A third challenge is converting
efforts spent on teaching usability testing into a tangible product that
can be measured by an academic institution. Ongoing efforts as well as
solutions to these challenges will be presented.

**REFERENCES**

Claus P.L., Gibbons P.S., Kaihoi B.H., and Mathiowetz, M. (1997).
Usability lab: A new tool for process analysis at the Mayo Clinic. In:
*HIMSS Proceedings.* Chicago: Healthcare Information Management Systems
Society. Vol. 2, pp. 149-159

Espinoza, J. (2001). Human factors in the engineered environment. In
proceedings "Enhancing Working Conditions and Patient Safety: Best
Practices. Pittsburgh, PA. Washington, DC: Agency for Healthcare
Research and Quality. \[Online\]
[www.quic.gov/workforce/Enhance/index.htm](http://www.quic.gov/workforce/Enhance/index.htm)

Gosbee, J.W. (1999). Human factors engineering is the basis for a
practical error-in-medicine curriculum. In proceedings of the *First
Workshop on Human Error in Clinical Systems*. Glasgow, Scotland:
University of Glasgow. [\[Online\]
www.dcs.gla.ac.uk/\~johnson/papers/HECS_99/Gosbee.htm](http://www.dcs.gla.ac.uk/~johnson/papers/HECS_99/Gosbee.htm)

Gosbee, J.W. and Lin, L. (2001). The role of human factors engineering
in medical device and medical system errors. In Charles Vincent (ed.)
*Clinical Risk Management: Enhancing Patient Safety*, 2^nd^ Edition.
London: BMJ Press.

Kohn, L.T., Corrigan, J.M., and Donaldson, M.S. (1999). *To Err is
Human: Building a Safer Health System*. Washington, DC: National Academy
Press. Pp. 140-149. \[Online\]
[www.nap.edu/readingroom](http://www.nap.edu/readingroom)

Sojourner, R.J., Aretz, A.J., Vance, K.M. (1993). Teaching an
Introductory Course in Human Factors Engineering: A Successful Learning
Experience. In *Proceedings of the Human Factors and Ergonomics Society
33^rd^ Annual Meeting.* Santa Monica, CA: Human Factors and Ergonomics
Society.

Welch, D.L. (1998). Human factors in the health care facility.
*Biomedical Instrumentation and Technology*. March/April 1998, 311-316.

Wiklund, M. (1995). *Medical Device and Equipment Design*. Buffalo
Grove, IL: Interpharm Press.