`

Technical Evaluation, Testing, and Validation of the 

Usability of Electronic Health Records: 

Empirically Based Use Cases for Validating SafetyEnhanced Usability and Guidelines for Standardization 

Svetlana Z. Lowry

Mala Ramaiah

Sheryl Taylor

Emily S. Patterson

Sandra Spickard Prettyman

Debora Simmons

David Brick

Paul Latkany

Michael C. Gibbons

This publication is available free of charge from:

http://dx.doi.org/10.6028/NIST.IR.7804-1

NISTIR 7804-1 
Technical Evaluation, Testing, and Validation of the Usability 

of Electronic Health Records: 

Empirically Based Use Cases for Validating Safety-Enhanced 

Usability and Guidelines for Standardization 

Svetlana Z. Lowry

Mala Ramaiah

Information Access Division

Information Technology Laboratory

Sheryl Taylor

Software and Systems Division

Information Technology Laboratory

Emily S. Patterson

The Ohio State University

Columbus, OH

Sandra Spickard Prettyman

Akron University

Akron, OH

Debora Simmons

St. Luke’s Health System

Houston, TX

David Brick

NYU Langone Medical Center

New York, NY

Paul Latkany

New York Eye and Ear Infirmary

New York, NY

Michael C. Gibbons

The Johns Hopkins University

Baltimore, MD

This publication is available free of charge from:

http://dx.doi.org/10.6028/NIST.IR.7804-1 

U.S. Department of Commerce 

Penny Pritzker, Secretary 

National Institute of Standards and Technology 

Willie May, Acting Under Secretary of Commerce for Standards and Technology and Acting Director 

NISTIR 7804-1 

October 07, 2015
ii

Acknowledgments

The authors gratefully acknowledge the intellectual contributions from the following:

• Mary Theofanos, M.S., Human Factors Scientist, National Institute of Standards and 

Technology

• Brian Stanton, M.S., Cognitive Scientist, National Institute of Standards and Technology

• Kristen Greene, Ph.D., Cognitive Scientist, National Institute of Standards and Technology

• David R. Hunt, M.D., Medical Director, Patient Safety & Health IT Adoption at Office of the 

National Coordinator, Office of the Chief Medical Officer

• Teresa Zayas-Caban, Ph.D., Special Assistant to the Deputy National Coordinator , Office of 

the National Coordinator

• Ellen V Makar, M.S.N., R.N.-B.C., C.C.M., C.P.H.I.M.S., C.E.N.P., US Dept. of Health and 

Human Services, Agency for Healthcare Research and Quality, Center for Evidence and 

Practice Improvement

• Michael L. Hodgkins, M.D., M.P.H., Chief Medical Information Officer, American Medical 

Association 

• Tejal Gandhi, M.D., M.P.H., CPPS, President, National Patient Safety Foundation

• Limor Hochberg, M.S., Human Factors Specialist, Life & Health, UL LLC, Wiklund R&D

• Kelly Cochran, M.S., R.N., Policy Advisor, Health Information Technology, Department of 

Health Policy, American Nurses Association

• Mary D. Patterson, M.D., M.Ed., Associate Vice Chair, Medical Education Research in 

Simulation, Director of Children's Academy of Pediatric Educators, Children's National 

Medical Center

• Janey Barnes, Ph.D., Human Factors Specialist, User-View, Inc.

• Susan Harkness Regli, Ph.D., Human Factors Scientist, University of Pennsylvania Health 

System

• Ross Teague, Ph.D., Senior Manager of User Experience, Allscripts Healthcare Solutions, 

Inc.

• Hardeep Singh, M.D., MPH, Chief, Health Policy, Quality and Informatics Program, 

Houston Veterans Affairs Health Services Research Center for Innovations 

• John Ritter, HL7 EHR Work Group co-chair; PHR Work Group co-facilitator; EHR System 

Usability Work Group co-facilitator HL7 International Mentoring Committee co-chair ISO 

TC/215 U.S. Technical Advisory Group member Western Pennsylvania HIMSS Board of 

Director member

• Thomas Elmquist, M.D., Cardiologist, New York Cardiovascular Associates

• Todd C. Hughes, Ph.D., Chief Technology Officer, Next Century Corporation

• Theresa Cullen, M.D., Chief Medical Information Officer, Veterans Health Administration 

(VHA)

• Jorge A. Ferrer M.D., M.B.A., Biomedical Informatician, VHA OIA Health Informatics, 

Adjunct Assistant Professor, UTHealth School of Biomedical Informatics

• Jan Horsky, Ph.D., Senior Research Scientist, Brigham and Women’s Hospital, Division of 

General Internal Medicine and Primary care; Harvard Medical School
iii

In addition, the authors appreciate the access to the data collected under NIST contract number 

SB134110CN0107 (UL-Wiklund) and the many instructive contributions from a broad spectrum of 

stakeholders on the draft document during an invited stakeholder meeting held at the National 

Institute of Standards and Technology on September 28, 2015, as well as peer reviewers for their 

constructive feedback on prior drafts of the document.

DISCLAIMER

Certain commercial entities, equipment, or material may be identified in this document in 

order to describe a concept adequately. Such identification is not intended to imply 

recommendation or endorsement by the National Institute of Standards and Technology, nor 

is it intended to imply that these entities, materials, or equipment are necessarily the best 

available for the purpose.
iv

Table of Contents

Acknowledgments........................................................................................................................... ii

Table of Contents.......................................................................................................................... iv

Executive Summary ........................................................................................................................v

1 Introduction: Enhancing Safety with Empirically Derived Guidelines for Standardization .. 1

2 Methods for Identifying Critical Use Risk Areas....................................................................... 2

3 Major Findings from Data Analysis .......................................................................................... 9

4 Methods for Generating Guidelines and Validation Use Cases.............................................. 15

5 Use Cases for Validation Testing to Ensure Safety-Enhanced Design .... Error! Bookmark not 

defined.

6 Empirically Based Human Factors Guidance for Safety-Enhanced Design of Health 

Information Technology ......................................................................................................... 36

7 Conclusions............................................................................................................................... 41

8 References ................................................................................................................................. 43
v

Executive Summary

This document provides the empirical rationale for critical patient safety-related usability guidelines 

for standardization. As well as requirements for validation testing to ensure safety-enhanced design. 

These standardization guidelines are targeted at eliminating ‘never events’ and associated patient 

harm by proactively addressing and mitigating the root causes of use errors from EHR design and 

implementation elements, as characterized in our framework on the relationship between usability 

and patient safety (NISTIR 7804). 

Requirements for validation testing are instantiated through realistic use cases that can be applied 

during design and evaluation of Electronic Health Record (EHR) systems and for user performance 

testing. The ultimate goal is to drive and empower effective and safe human performance in the use 

of EHRs. The objective of this research is to enhance safety-related usability with empirically

derived guidance in order to improve the effectiveness and efficiency of EHRs by eliminating or 

reducing the most critical and likely causes of patient harm from mistakes and errors in interaction. 

This research drew upon five different methods of empirical human performance data collection, 

using crosscutting analytic methods, with a diverse set of analysts from different disciplines, 

backgrounds, and perspectives. Researchers applied this mixed method approach in order to capture 

user expectations, knowledge, and outcomes regarding EHRs. Multiple forms of data were collected 

from a variety of user types, allowing for a comprehensive view of EHRs. Two large, multi-hospital 

healthcare systems in the U.S. served as sites for most of the data collection, including observations 

and interviews. Data collection included: 1) an online survey; 2) site observations; 3) follow-up 

interviews with users; 4) usability testing of five different EHRs; and 5) expert reviews of the same 

EHRs. Research results demonstrate strong congruence among the data, methods, and the analysts. 

Human factors guidelines for standardization, which were explicitly derived from the empirical 

evidence obtained through field data collection, are provided to improve the safety-related usability 

of EHRs in each of the following three critical use risk areas:

1) Consistently display information critical to patient identification in a reserved area to 

avoid wrong patient errors, 

2) Provide cues to reduce the risk of entering information and writing orders in the wrong 

patient’s chart, and 

3) Support efficient and easy identification of inaccurate, outdated, or inappropriate items in 

lists of grouped information by having information presented clearly and in a well-organized 

manner.

Two use cases, one for the inpatient setting and one for the outpatient setting, are provided for 

validation with summative usability testing. Employing use cases oriented to empirically- derived 

usability challenges is intended to validate that potential patient safety risks are proactively mitigated 

in the tested EHRs. These use cases may also prove useful for organizations for adapting aspects of 

the design during implementation and optimization processes to protect against inadvertently 

introducing new risks to patients.
vi

Ultimately, this research demonstrates that patient safety is negatively affected when critical safety 

tasks are performed with the support of poorly designed EHRs. As a result, mistakes and errors 

frequently occur; with end users becoming frustrated and unwilling to trust the systems they are 

given and therefore are more likely to rely on potentially unsafe workarounds.
1

1 Introduction: Enhancing Safety with Empirically Derived 

Guidelines for Standardization

This document outlines the empirical rationale for critical patient safety requirements instantiated in 

use cases that can be used during the design, evaluation, and user performance testing of EHR 

systems. Electronic Health Records (EHRs) offer great promise for improving healthcare processes 

and outcomes, including increased patient safety. Emerging evidence suggests that the use of health 

information technology (HIT) may help address significant challenges related to healthcare delivery 

and patient outcomes.

1 For example, three reports suggest that the use of HIT may improve 

healthcare outcomes2 and reduce patient mortality.3 In addition, the use of HIT is a key component 

of a national strategy to improve healthcare quality and patient safety.4 Given the estimate that one 

in three patients will potentially be harmed during a hospitalization,5 the role for EHRs to improve 

patient safety may be significant. 

On the other hand, studies found that patient mortality unexpectedly increased following the 

introduction of an EHR in a pediatric hospital6 and that an order-entry system contributed to a severe 

medication overdose.7 Therefore, empirically-derived guidelines for standardization are needed to 

foster enhanced ability to protect patients from inadvertent harm from EHRs. As with any HIT, EHR 

usability problems that can adversely impact patient safety can be assessed, understood and 

managed. Data collected through multiple methods can increase the validity of findings regarding 

the magnitude of risk to patients.

The ultimate objective of enhancing safety with empirically derived guidelines for standardization is 

to improve the effectiveness and efficiency of EHRs, and thus reduce the opportunity for patient 

harm from mistakes and errors. Failing to have effective and efficient EHRs could result in clinical 

users ‘extensive workarounds that can directly and negatively impact patient safety. 
2

2 Methods for Identifying Critical Use Risk Areas

Researchers applied a multiple research methods approach in order to triangulate user interactions,

experience and dispositions of common encounters with EHRs. Multiple forms of data were

collected from a variety of user types allowing for a comprehensive view of EHRs. Data include: 1)

an online survey; 2) site observations at two geographical locations; 3) interviews with users at these

two locations; 4) usability tests with five different EHRs; 5) and expert reviews of the same EHRs. 

A team of researchers with expertise and extensive experience in research methods executed the

effort, aided by three additional researchers, including clinical experts and human factors experts.

Data Collection

Data for this study were collected via NIST Contract Number SB134110CN0107 (UL-Wiklund). 

Data collection commenced in September 2010 and concluded in May 2015. Two large, multihospital healthcare systems in the U.S. served as sites for much of the data collection, including the

observations and interviews. Two weeks of observations and 40 interviews occurred in a variety of

care settings and with different types of user groups who routinely used EHRs. There was extensive

effort of unobtrusive ‘in situ’ observation and follow-up interviews in these locations. A total of 86

EHR users were interviewed and observed, including 49 medical assistants and administrative staff

members. Verbatim transcriptions of interviews and researcher field notes provided accurate and

reliable data for the analysis. There were over 300 pages of interview transcripts and 336

observation notations.

In addition, five EHR developers volunteered their EHRs for usability testing. There were a total of 

15 two-hour test sessions per EHR, resulting in a total of 63 participants, including 31 providers 

(physicians, nurse practitioners, and physician assistants), 21 nurses, and 11 medical assistants. 

Several two-person teams, each including a test administrator and a data analyst (i.e., data logger), 

conducted the usability tests. All test sessions took place in usability test facilities that included a test 

room and adjacent observation room. All test sessions were video recorded. Test data included use 

errors, “close calls”, difficulties, anecdotal comments about EHR interactions and quantitative 

ratings of the EHR according to a scoring system focused on safety-related usability which was 

developed as part of NIST contract SB134110CN0107. This scoring system, using responses as a 5-

point Likert scale from 1 = strongly disagree to 5 = strongly agree, was comprised of these 

statements:

• The EHR uses familiar terms, symbols, and units.

• The EHR makes important information stand out.

• The EHR’s workflows match my expectations.

• The EHR makes it easy to detect and correct mistakes.

• The EHR gives me a comprehensive view of a patient’s health.

• The EHR will prevent harmful mistakes.

• The EHR makes it easy to share information with other healthcare professionals.

• I can use the EHR effectively while working under pressure.

• The EHR is easy to learn to use.

• I am confident that I can use the EHR to perform tasks correctly.
3

At the same time, six usability, informatics, and human factors experts independently reviewed the

same five EHRs in order to identify interface strengths and weaknesses of each. Of 665 strengths

and weaknesses identified overall, 60 weaknesses were identified by a human factors expert to

potentially have patient safety implications. Examples include having the patient name displayed on

the lower right corner of the screen, unlabeled graph axes, values displayed without accompanying

units (e.g., 225 without lbs), a value of 11 years and 11 months plotted as 11 years on the X axis of a

graph, not allowing physician users to edit or cancel an order after submission, and requiring one tab

to be open when writing an order and a different tab to be open when submitting a medication order.

Finally, a single academic medical center participated in an online survey, with approximately 2,500

surveys sent to clinicians and administrators from 19 different departments/units. There were 559

responses, of which 339 responses were complete (13.5% response rate). Nurses and physician

providers were the largest categories of user types who responded. There were 21 questions on the

survey, including two open-ended questions: 1) What EHR characteristics make it particularly easy

to use? and 2) What EHR characteristics make it particularly difficult to use?

Data Analysis

Quantitative and quantitative data set/analysis are described in NIST GCR 15-996 “Technical Basis

for User Interface Design of Health IT”.

1 Consecutive qualitative analysis of the interview and

observation data began with multiple readings of the full data set by the lead researcher and one of

the research analysts. Each of these two researchers coded a subset of the qualitative data in order to

begin a conscious and collaborative process of peer debriefing related to coding agreement. The

goal was to insure that the researchers’ use of codes and their application to segments of text was

consistent; “the more coders (using the same codebook) agree on the coding of a text, the more we

can consider the codebook a reliable instrument”8 (page 310). This process is recommended in

qualitative research in order to establish the “credibility” or “truth value” of the findings.

9,10 During

peer debriefing sessions, researchers operationalized all codes and reviewed the data linked to

various codes. This process occurred regularly during the multiple coding phases, allowing for

continual reflection on and refinement of the codes and their application to the data.

Both researchers continued to read through the interview and observation data, creating tentative

labels for chunks of data that summarized what was emerging from the data. This was the open

coding process11 or first cycle coding12. During this coding cycle, approximately 60 different codes

were used, with 30 of those occurring consistently across the data. While many of the remaining 30

codes only occurred once or twice, several of these represented critical cases where the data seemed

important enough (for example, those related specifically to patient safety). These critical case

codes became part of the code list that was used in the next coding cycle. 

Once saturation occurred in the open coding process, the place where no new properties or

dimensions emerged from the coding process, we moved on to axial coding. Axial, or second

cycle13, coding provided a means to identify the relationships and connections amongst the open

codes. This allowed for the development of conceptual categories that link the codes and associated

data. Richards and Morse14 argue that coding “leads you from the data to the idea, and from the idea

 1 Available at http://nvlpubs.nist.gov/nistpubs/gcr/2015/NIST.GCR.15-996.pdf. Accessed September 28, 2015.
4

to all the data pertaining to that idea” (page 137). During this phase of the analysis, two top-level

categories of codes: 1) issues related to input and/or handling of data and information and 2) issues

related to understanding and use of the EHRs were identified. 

Finally, selective coding identified core variables and emergent themes that occurred across the data

set.

15 Each researcher developed and shared memos and models about ongoing interpretations of the

data as a means to identify conceptual categories and move toward the development of emergent

themes for critical use risk areas that allowed researchers to predict patterns “of what may be

observed and what may happen in similar present and future contexts.”16 Here, three major critical

use risk areas emerged as our ultimate findings: Identification of information, Consistency of

information, and Integrity of information. 

At the same time, the lead researcher and the data analyst coded the usability test data. The same

conceptual categories and emergent themes developed in this data as in the interview and

observation data. Similarly, the lead researcher reviewed the survey data for overlap between this

and other data sources. Again, similar ideas surfaced across the data sets. Finally, the human factors

researcher examined expert reviews to discern patterns across reviews related to the issues under

consideration, specifically the relationship between the EHRs and patient safety. The lead

researcher used the findings to map back to the other data forms. The conceptual categories and

emergent themes are represented across the data sets, demonstrating the strength of the findings. 

Examples of Codes, Categories, and Critical Use Risk Areas

In qualitative research, a code “symbolically assigns a summative, salient, essence-capturing,

and/or evocative attribute for a portion of language-based or visual data.”17 Coding is a cyclical act

that serves as an intermediary between data collection and data analysis. Saldaña18 argues that

qualitative codes capture essential elements of the data, and when those elements form clusters they

can “facilitate the development of categories and thus analysis of their connections” (p. 8). Codes

are rooted in the data, but as coding and analysis continue, codes and categories become more

refined, and often more conceptual and abstract. A further shift is evidenced “when the major

categories are compared with each other and consolidated in various ways, you begin to transcend

the “reality” of your data and progress toward the thematic, conceptual, and theoretical” (p. 12). 

Below are excerpts of particular pieces of interview data and the codes applied to them, followed by

how these led to broader categories of analysis and later to emergent themes that ran across data sets.

It should be noted that the excerpts below are exemplars to demonstrate how data were coded,

however many more data segments were coded with the same codes. Many codes resurfaced

multiple times in the same interview and most occurred across all interviews.
5

Table 1. Examples of individual data excerpts and codes

DATA CODE(S)

“Sometimes, I might issue a rapid strep test and then save the note as a draft so that I have to go back in at end of day to save as

final based on strep test results. I might need to edit [the] plan, and then I also give patient both plans of action. [The] system will

send me a message reminding me to save the drafts, but you have to be paying attention, and there are a lot of things to pay

attention to”.

DRAFT VS. FINAL;

RELIANCE ON

MEMORY

“Running through a list of med[ication]s can take quite a bit of time. When a patient gets admitted to the hospital and discharged,

the med[ication] list changes. To me, there is a right way to write a prescription, so that we can communicate it. When [a] patient

gets discharged, sometimes pertinent medic[ations] can get deleted, and the format changes. When they get discharged, the units

might be off, the units might say mg, but the mom tells me she gives mL. The mom won’t know what mg means. If [the] patient

come[s] from [the] hospital, it might take me a half hour to update it”.

MEDICATION LISTS

CHANGE;

MEDICATIONS

DELETED;

UNITS OF MEASURE

CHANGE

“I think you could send a message on the wrong patient. Let’s say that the mom called and the child has breathing difficulty and

you are accidentally in [the] wrong record. It’s not the information that gets into the wrong patient, it’s knowing what patient the

information belonged to. Sometimes, [the] wrong vaccine gets recorded. I notice something might be wrong, the age is wrong, I

can get someone to go in and delete [the] vaccine, but then we are missing the information on [the] proper patient’s record. We

may be able to identify later when that patient comes in, but if we can’t find it, the patient gets an extra vaccine”.

WRONG

PATIENT/CHART;

VACCINE ERRORS

“My patient list is incorrect. It is not a complete patient list and I don’t know why. Sometime over the past 6 to 8 months 2/3 of

my list got purged. I have to remember the patients’ names and search through [the current EHR] to find...It is very frustrating

because I have no idea why it happened. Suddenly, one day I had 3,200 patients on my list and the next day I had 700. It doesn’t

make any sense. Every day the system does not feel firm or complete”.

WRONG PATIENT

LISTS;

FRUSTRATION

“A patient going from outpatient to inpatient or vice versa. Often times the medications are not addressed appropriately at those

transition periods. Often what’s in the discharge summary is not what’s in the medication list in the patient summary and so often

times it’s confusing to figure out. The person discharging the person is not diligent in updating the medications and I don’t know

if this happens, but it seems to be that somehow the inpatient medications and discharge summary medications should all autopopulate”.

WRONG/DISCREPANT

MEDICATION LIST;

NEED FOR AUTOPOPULATION

“No checks and balances for diagnoses in patient summary. Say a patient is seen by one provider and provider goes in patient

summary, sees that they have [a] scar on [the] chest so must have had CAD [Coronary Artery Disease] and surgery. If I see the

patient and change the summary all around and correct it, say they never had a heart attack, say I update that, if that patient goes to

see someone else... there’s no accountability. Anyone can change anyone else’s patient summary. Patient could come back to me

after seeing 3 other people, no longer looks like what I spent an hour updating. I can go back and look at my patient summary and

try to paste it forward. But [there] are oftentimes disease-specific information that other services might delete, and they delete it,

but it’s actually the record. Can be frustrating and lead to errors”.

CHANGES MADE TO

NOTES;

NO

ACCOUNTABILITY;

DELETED

INFORMATION;

FRUSTRATION;

ERRORS
6

Table 1. Examples of individual data excerpts and codes (Cont.)

DATA CODE(S)

“Patients can receive 2 doses of medication because of how read-only scripts are handled in the system, e.g., [for example] readonly means that there is no physical order, the pharmacy can view and dispense the medication, but with a read-only order there is

no way to scan or manually document that the patient received the medication. Order could be filled and given twice. EHR user

has to complete 8-9 clicks to get to a more detailed view to see comments if the script has been given to the patient”.

WRONG

MEDICATION OR

DOSE;

DIFFICULTY OF USE

(8—9 CLICKS)

“I’ve seen this EHR [at this hospital] go from a reporting mechanism to a data entry mechanism, to the point where it becomes a

file cabinet that you shove paper into. I guess if you control it [data entry], you know where things are. But, if my assistant or my

colleagues begin to use my file cabinet, pretty soon I’m not going to know what’s in there. I might be able to find a paper I filed 6

months ago, but it’s a needle in a haystack. Indexing is not standardized. It’s difficult to figure out where to start. [This is] what

am I worried about”.

“FILE CABINET”;

OUT OF CONTROL;

“NEEDLE IN

HAYSTACK”;

NEED FOR

STANDARDIZATION

“The list in the EHR is populated by multiple individuals. It’s a mish mash. Stuffing information in drawer, I may own a section

of that, not [the] entire piece”.

“MISH MASH” OF

INFORMATION

USED BY DIFFERENT

INDIVIDUALS;

POSSBILITY FOR

CHANGES/ERRORS

“Let’s say the patient is on Coumadin and then an internist gives an antibiotic. If [I] get prompted to refill the Coumadin, [then] I

get prompted that [the] patient is on an antibiotic and it tells me to consider decreasing [the] dose. That’s a robust decision support

piece that only works with providers in this EHR system [at this hospital]. It only works if I’m here and I see that you prescribed

that. It [EHR] doesn’t reflect the antibiotic received at [the] walk-in clinic”. 

WRONG

MEDICATION OR

DOSE;

MEDICATION

INTERACTIONS

“Omitted information. It would be through omission, or you chart on the wrong kid. That would be human error. Safety thing is

when patient was omitted and all information was left out. The log in/log out thing is a huge pain in the butt. If I’m working at the

desk and someone says “Hey,” I run. If User Y comes back and was working at that spot, he might sit down and chart on me [the

chart I was working on before leaving]”.

OMITTED

INFORMATION;

WRONG

PATIENT/CHART
7

As coding continued, codes seemed to cluster into two major conceptual categories: 1) input and

handling issues and 2) understanding and use issues. For example, “Draft vs. Final” seemed to

be an input issue, as did “Changes Made to Notes” and “File Cabinet.” These codes were

connected by their relationship to the ways in which data made its way (or not) into the system. 

Likewise, data related to “Medications Change” or “Wrong Patient” clustered around the ways in

which participants understood and used the EHRs. Sometimes chunks of data related to both

categories, as in the case of “Wrong Patient/Chart.” For example, P43 above noted that someone

might be working on a chart, need to leave for another task, and then someone else would begin

to work on it, thinking it was a different chart. This was an input issue (in that data was input

into the wrong chart), but also a use issue in that it was often difficult to discern which chart one

was in at a given time. The second cycle of coding allowed analysts to revisit previously coded

data and cluster it around these two conceptual categories. During this process, analysts noted

that both categories produced issues related to Identification of Information, Consistency of

Information, and Integrity of Information. These critical use risk areas pulled together a variety

of related codes that were united conceptually. For example, Identification of Information

represented those instances in the data where a piece of information was unable to be identified,

was misidentified, or was difficult to identify (this could be a patient name, a medication, a chart

number, or whether something was a draft versus a final entry). Consistency of Information

related to where and how information was presented, and often referred to a lack of

standardization in where and how information was presented as in where and how the patient

name was presented or a medication was entered. Finally, Integrity of Information encompassed

those codes where the accuracy of information was difficult to discern, sometimes due to

information being changed, being deleted, or not being entered. 

Once coding of observation and interview data was complete, researchers looked across data sets

for congruence of analysis. Many of the codes generated in the analysis of observation and

interview data also emerged in the usability tests and expert reviews. For example, location of

patient name, unlabeled graph axes, values displayed without accompanying units, not allowing

physician users to edit or cancel an order after submission, and requiring one tab to be open

when writing a medication order and a different tab to be open when submitting a medication

order. Survey data also corroborated the coding structure and the subsequent development of

conceptual categories and themes. Identification of information, Consistency of information, and

Integrity of information cut across the data as major areas of use risk.

Trustworthiness

According to Lincoln and Guba9

, trustworthiness involves establishing credibility,

transferability, dependability, and confirmability. Triangulation is one way to insure a

comprehensive, well developed, and systematic analysis that provides for credibility in the

process and the findings.

19,20 Triangulation can take many forms, including triangulation of

methods (using multiple methods for both data collection and analysis), triangulation of sources

(from within the same method—for example having Medical Assistants, Nurses, and Providers

as participants in the interviews and usability tests), and triangulation of analysts (to provide for

differing analytic lenses and perspectives). Researchers in this project used all three types of

triangulation to provide a more robust process and insure a more holistic and comprehensive

picture of the findings. Mays and Pope21 suggest additional mechanisms for enhancing the
8

trustworthiness of a study, including respondent validation, where study participants are asked to

respond to and assess the researchers’ interpretations. While traditional respondent validation

(or member checking as it is also known) was not utilized here, researchers held a debriefing

session with people from a range of roles and positions, including nurses, doctors, and human

factors experts in order to obtain feedback regarding the analysis, interpretation, and findings.

Peer debriefing and respondent validation provided additional credibility for the study.

9

The use of “thick description”9 and the voices of participants allow for greater transferability, or

the ability of readers to determine for themselves whether or not the results generalize to their

own situations. The presentation of extensive quotes and the detailed description of the coding

and analysis process in this report allow for such transferability. Dependability refers to how

likely it is that the findings would be consistent if the study were done again with a similar set of

participants in a similar context. Dependability can be achieved through the use of an external

audit where researchers who were not involved in the project review the process and products of

the research. In this study, the debriefing session with a variety of different experts offered a

space for presentation and discussion of the research process and the findings. Confirmability

provides a mechanism to determine if the conclusions, recommendations, and/or interpretations

can be traced to their sources, insuring that the findings are a product of the study and not of

researcher bias. An audit trail of the data and its processing allows for such confirmability. 

Mays and Pope20 also argue that a clear presentation of the data collection and analysis methods

provide for greater validity of the study. 
9

3 Major Findings from Data Analysis

This research examined the use of EHRs in different healthcare settings and with different types 

of users in order to gain a comprehensive view of the EHR impact. The objective is to improve 

their effectiveness and efficiency and thus reduce the likelihood of contributors to avoidable 

patient harm. We found three overarching themes representing critical areas of use risk, based 

upon a convergence of triangulated evidence from all of the data sources: identification of 

information, consistency of information and integrity of information. Within each overarching 

risk area are subcategories, characterized by prototypical statements and questions, and possible 

consequences, all rooted in the data. These are detailed in Table 2 below. Across all three of the 

risk areas and their related subcategories there arose several major issues related to patient 

safety: 1) the occurrence of unintended actions; 2) the likelihood of use errors; and 3) the high 

level of user frustration. The three areas and their related issues highlight a lack of effectiveness 

and efficiency related to EHRs. Ultimately, both the objective (actual observation and analysis 

of user performance) and subjective data demonstrate that these issues negatively affected patient 

safety during safety-critical tasks and times. For example, one participant discussed transition 

periods and how these often led to discrepancies in medication lists:

“So the patient going from outpatient to inpatient, or inpatient to outpatient. Often times 

the medications are not addressed appropriately at those transition periods, and so often 

times what's in the discharge summary is not what's in the medication list in the patient 

summary...And so often times it's confusing to figure out...The patient can end up being 

confused and taking either too much or too little, or the wrong medication combination. 

I, as the provider, on the other hand am unclear on what the patient is actually taking, 

and it leads to confusion when the patient calls and says, "I'm not responding to the 

treatment in the hospital. Then I have to go back and try to figure out, okay, are they not 

responding because they're not actually taking the right medicine, or something like that? 

And it also leads to confusion because the-- I had one patient who called our office from 

their hospital bed and said, "They're sending me home. I'm confused about my 

medication and nobody will talk to me." So not only was the documentation fraught with 

errors in instruction to the patient, they were just as confused as I was trying to figure out 

what they were going home with.”

The data excerpt above demonstrates how the integrity of the information is compromised 

leading to incorrect dosages or medication combinations. Effectiveness is diminished as the 

likelihood for critical use errors increases and efficiency is affected since it will now take 

additional time to identify accurate information. This relationship was seen many times across 

the data set. Also related to integrity of information, a different participant noted how difficult it 

was to create an accurate patient summary:

“Actually, I was in with a patient. Gosh, was it Monday? I was trying to update the 

patient summary because that's essentially what I spend a lot of my time doing now with 

all new patients is I'm building all the patient summaries. Let's see if I can get to this 

patient summary. What ended up happening is, as I'm updating, I can hit update here, 

and when I'm updating, I can't see labs, I can't see anything else. I'm asking them, 

"When's your last tetanus? Have you had your diabetes screen checked?" And I can go 

up here to Actions, which I've now learned, and I can open up any one of these things in 

a different-- let's see if we can get the labs. I can open up labs, but what happened to me 
10

last time was that I was trying to open these up and it overwrote whatever I was doing 

here. It wouldn't let me save anything. It wouldn't let me get back to it to save it.”

Not being able to see lab results and other pertinent information, as well as not being able to save 

information as needed also reduce effectiveness and efficiency and create a situation where 

frustration and unintended actions are likely to occur.

In addition to the integrity of information, the consistency of information presentation 

and where to find it was a common issue. Two other participants articulated examples that many 

others corroborated related to this pattern:

“I’ve seen this EHR here at [hospital] go from a reporting mechanism to a data entry 

mechanism, to the point where it becomes a file cabinet that you shove paper into. I guess 

if you control it, you know where things are. But, if my assistant or my colleagues begin 

to use my file cabinet, pretty soon I’m not going to know what’s in there. I might be able 

to find a paper I filed 6 months ago, but it’s a needle in a haystack. Indexing is not 

standardized. It’s difficult to figure out where to start...

Because if you read through it and you don't change it to make it up to date, and you 

don't bring forward the information that you've learned - they've got a new allergy, or 

they used a medicine that you gave last time and they have an adverse affect, or anything 

that they've told you that happened since you last used that note. If you don't bring it in to 

that note, and you don't record it, then not only have you created a safety concern for 

that patient, you've created a liability for yourself and for the institution.”

Finally, the identification of information arose as a major issue, with many participants 

noting how difficult it was to easily locate and identify information and how easy it was to be in 

the wrong place. The data excerpt from a different participant below clearly illustrates the 

potential consequences of this issue:

“An example today: When I went in today to see a patient, there were two notes saved, 

one was a 3 year old draft that my nurse started, the other note was for a different 

patient. That’s easily solved. I went in the room, noticed an incorrect note. I sent a 

message to the nurses. Asked them to delete it from record. But, in the meantime, one of 

my staff members went into room of 12 month old and couldn’t find the record. I think 

you could send a message on the wrong patient. Let’s say that the mom called and the 

child had breathing difficulty and you are accidentally in wrong record. It’s not the 

information that gets into the wrong patient, it’s knowing what patient the information 

belonged to. Sometimes, [the] wrong vaccine gets recorded. I notice something might be 

wrong, the age is wrong, I can get someone to go in and delete the vaccine, but then we 

are missing the information on the proper patient’s record. We may be able to identify 

later when that patient comes in, but if we can’t find it, the patient gets an extra vaccine.”

It is important to note that the examples presented above serve as exemplars for a broad range of 

data found across the different methods that support the use risk areas and subcategories. These 

are by no means unique statements, but representative of the data as a whole.
11

Table 2. Summary of Analytic Findings on Major Safety-Related Risk Areas and Possible Consequences

Areas of Critical Use Risk Subcategories Possible Consequences

Identification of Information

Am I in the right place and doing the right 

thing? 

• For patient

• For patient lists

Incorrect patient list

Who are these patients? 

• Wrong treatment

• Wrong billing

• Wrong charting of information

• Missed, omitted, delayed care

• Care or billing activity conducted on the wrong patient

• Have to pull up every patient chart

• Wrong medication ordered

• For records

• For medication/order

Passing/sharing information

What happens in the handoff?

Multiple EHRs used

What happens when EHRs don’t coordinate?

• Data/information are not recorded in EHR

• Misrecording or recording in wrong patient chart

Fragmented information

Data are often fragmented and found in multiple 

places

• Often no context for displayed information

Consistency of Information

Why are things not listed and displayed in 

standardized ways? 

Misidentified patient/chart Where am I?

Record number is incorrect/Patient name is 

misidentified. 

• Documentation/orders in wrong chart (often without 

knowing it)

• For information

• For organization

• For format

• For different systems

• For draft vs. final versions

• For omissions and/or changes

Supplements used

Did I remember to transfer data to the EHR? 

Paper, whiteboards frequently used (reliance on 

memory)

Multiple EHRs used

Where do I find X on this EHR?

Functions and screens shift Where is my 

information?

• Cannot find information when needed in the EHR

• Reliance on memory for transfer of information/data

Cannot find information

Where is my information? Am I in the right 

place?

• On screen

• In file

• In EHR

• Functions and screens shift

• Information found in different places (including record 

number, patient name, medications prescribed, etc.)

Standardization of where things are and what 

they are

• Location and format of date

• Location of record number

• Location and format of name

• Format of amount (e.g., metric vs. US)

• Running list of current medications

• Notation in wrong record

• Incorrect diagnosis or prescription

• Incorrect medication or order OR double vaccine
12

Table 2 Summary of Analytic Findings on Major Safety-Related Risk Areas and Possible Consequences (Cont.)

Areas of Critical Use Risk Subcategories Possible Consequences

Integrity of Information

Why and how are things changed, deleted, 

or omitted?

• Lack of control over changes in dates, 

notes, units of measure

• Inability to know what information is 

Cannot figure out EHRs

How do I do this?

• Navigation is difficult

• Adding/deleting data is difficult

• Scrolling through long notes is timeconsuming

• More likely to just use whiteboard/paper and not put 

data in EHR, resulting in incomplete files/charts

 valid, relevant, and up to date Draft vs. final version

Is this a draft or final version? Often forget to 

finish a final version

• Omissions of data/information; notation of incorrect 

data/information

• What happens if change in patient condition in interim?

Changes to note/chart

You changed what?!! 

• Different user can change someone else’s 

note/input

• Information lost

• Inaccurate and/or incomplete data and charts

Common references not there

Why are height and weight not here? (and other 

common data like vital signs)

• Having to do things (input) multiple times or search 

multiple places
13

Included in the two use cases are the following aspects, categorized by the areas of use risk that

emerged from the data analysis and related to supporting evidence from the data analysis:

1) Critical Use Risk Area 1: Identification of Information

A) Accidentally transposing a patient’s first and last name is easy to do when the 

names are both commonly used first names as well as possible last names

(e.g., William John, John William). This increases the potential for making 

clinical orders or documenting in the wrong patient’s chart due to the ease of 

inverting the names when searching for the chart, flipping the open chart to 

another person and mistakes can more easily occur when quickly scanning the 

name to confirm that the right chart is open. In the use cases for eligible 

providers and nurses, both William John and John William need to be 

included in the patient list.

B) Getting confused as to which chart is actively open for ordering medications 

and viewing information is challenging when there is an interruption in the 

midst of providing care for one patient in order to a high priority activity like 

look up the current medications for another patient in response to a phone call 

from the Emergency Department nurse for a patient who is not able to provide 

the information reliably.

C) Identifying an allergy to the common sugar substitute that is contained in the 

hospital’s standard diabetic diet is challenging when relying upon clinical 

decision support alerts that are not designed to cover this scenario and/or have 

high false alarm rates.

D) Being interrupted while providing care to one patient can increase the risks of 

performing clinical actions or documenting information in the wrong patient 

chart.

2) Critical Use Risk Area 2: Consistency of Information

A) Having a primary care provider in the outpatient setting identify that the 

results of a diagnostic (MRI imaging) test are not available because they were 

not ordered as planned during a prior hospital stay can be challenging when 

results from within an organization and from other organizations are not 

displayed and sorted in a consistent fashion.

B) Managing and documenting an allergy to a preservative in intramuscular 

vaccines, thimerosal, is challenging because it differs from how the typical 

allergy to a medication is identified and alerted with clinical decision support 

and visualization paradigms Clinical decision support recommends ordering a 

vaccination for influenza despite the patient having an allergy to thimerosol, 

which is a preservative in the vaccination in the traditional injection delivery

mechanism. A different form of the vaccine (that does not contain thimerosol) 

needs to be ordered to avoid an allergic reaction. 
14

C) Treating an infectious eye disease requires coordinating care with other 

specialized personnel, including infectious disease specialists and eye care 

specialists. It can be particularly challenging to coordinate care using “readonly” documentation.2

D) As soon as the nurse notices that the patient had a markedly pink irritated right 

eye on morning rounds, this information is critical to share quickly with other 

care providers in order to protect healthcare workers and other patients from 

infection by initiating and maintaining contact isolation precaution 

procedures. Sharing this information across transitions of care is similarly 

important.

3) Critical Use Risk Area 3: Integrity of Information

A) Managing medications requires reviewing medication lists which have 

discrepancies.3

B) Reviewing the laboratory results of a blood test cannot be done during the 

primary care provider outpatient clinic visit with the patient because the 

results are not available for review.

C) An imaging test that was not completed in the acute care setting is not readily 

discoverable during the follow-up visit with the primary care provider as it 

was not completed in the primary care setting.

 2 Supporting evidence from the interview data: “Patients can receive 2 doses of medication because of how read-only scripts are 

handled in the system (e.g., read-only means that there is no physical order, the pharmacy can view and dispense the medication, 

but with a read-only order there is no way to scan or manually document that the patient received the medication. Order could 

be filled and given twice. EHR [Electronic Health Record] user has to complete 8-9 clicks to get to a more detailed view to see 

comments if the script has been given to the patient.”

3 Supporting evidence from the interview data: “I guess part of the biggest thing that we noticed, or that I see, is the transition 

periods. So the patient going from outpatient to inpatient, or inpatient to outpatient. Often times the medications are not 

addressed appropriately at those transition periods, and so often times what's in the discharge summary is not what's in the 

medication list in the patient summary... And so often times it's confusing to figure out. The person discharging the patient is not 

diligent in updating the medications. And I don't know if this happens but it seems to me somehow the inpatient medications and 

discharge summary medications should all populate the updated patient summary, but I don't know if that happens. And so if 

there's a discrepancy between what's in the patient summary med-list versus what's in the discharge summary, what can happen 

as a result of that? The patient can end up being confused and taking either too much or too little, or the wrong medication 

combination.”
15

4 Methods for Generating Guidelines and Validation Use Cases

Group of the human factors experts collaboratively mapped the findings from the data analysis to 

human factors terminology in order to relate the insights to what is known in the human factors 

field about contributors and barriers to successful, high reliability, and high reliance performance 

by experts in domains with high consequences for failure. This translation resulted in 

transforming the major findings into human factors issues and deficiencies, which enabled a 

more targeted search for existing solutions and standards on which to base the recommendations. 

Subsequently, a team composed of a methodologist, clinical physicians, clinical nurses and 

human factors experts provided guidance on the generation of the recommendations and use case 

during a focused two-day working meeting. As a result of this meeting, the team generated 

safety-related usability technical guidance that draws upon existing human factors knowledge 

and is empirically grounded and prioritized on the basis of being repeatedly raised as important 

across five data collection methodologies.
16

5 Use Cases for Validation Testing to Ensure Safety-Enhanced 

Design

The research findings provide empirical evidence about critically important usability issues that 

could potentially have negative impacts on patient safety for providing clinical care with the 

support of EHRs in both the inpatient and outpatient care settings. These issues provide the 

foundation for the creation of two challenging use cases intended for summative usability testing 

validation.

Healthcare is delivered within a system encompassing both ambulatory and inpatient settings. As 

such, designing support for clinical care in either or both of these settings is challenging even for 

an ordinary evolution of the care of a typical patient over a select time period in his or her life 

that requires continuity of care across multiple providers with dedicated roles. 

In this section, we provide for evaluation purposes one use case for inpatient care and one use 

case for outpatient care. These two use cases are contextualized within an overarching narrative 

for a single patient with multiple care activities conducted by multiple types of care providers. 

When recruiting study participants for the validation testing, there are two categories of care 

providers, 1) eligible professionals (medical doctors, nurse practitioners, or physician assistants) 

and 2) nurses (registered nurses, licensed practical nurses). By employing the same patient for 

the validation testing, it is possible to reuse demographic information and test data.

Overview of both use cases

The two use cases are situated within an integrated series of care encounters, consisting of:

1. An acute care hospital admission through the emergency department of a 45-bed community 

hospital for a number of concerning symptoms, including chest pain and a productive cough.

2. An outpatient visit with a primary care provider two years later, after a recent hospital 

admission for a possible stroke
17

The use cases for evaluation cover selected aspects of the interactions dependent on the 

electronic health record. Initially, an overall scenario of the patient visit is provided. Next, the 

detailed steps involved by each user interacting with the EHR are enlisted. This is followed by a 

short description of the critical usability issues in the tasks of each of the user roles.

Use Case 1: First Hospital Stay

We discuss the use case in several parts. The complex scenario is likewise broken into relevant 

parts for better understanding and application. The scenario description is followed by the user 

tasks based on roles and the workflow per the scenario. Hence, the user may appear repeatedly as 

called by the events mentioned in the scenario. We associate the critical usability issues 

encountered in the performance of the tasks under each role as a subcategory. 

Scenario 1, Part I

Mr. William John is a 65-year-old Hispanic male who presents to the Community General 

Hospital Emergency Department with complaints of severe chest pain, difficulty breathing and 

productive cough. Mr. John has a known history of Type 2 diabetes mellitus and 

hypercholesterolemia. His allergies include Thiomerisol and “pink sweetener”. He is triaged and 

taken into the Emergency Department (ED). The ED nurse records the vital signs, which are

within the normal limits except for lower oxygen saturation of 90% and an increased heart rate 

of 110 beats per minute, and informs the Emergency Physician. The Emergency Physician orders 

oxygen therapy per nasal cannula immediately, 12 lead EKG, serial troponin levels, chest X-ray, 

sputum gram stain now, CBC, continuous telemetry monitoring, finger stick blood glucose 

following the hospitals s diabetic protocol and continuation on his routine medications of a long 

acting insulin, regular insulin on a sliding scale, atorvastatin, vitamins and aspirin. He also starts 

the patient on azithromycin. He orders an 1800 calorie diabetic diet. The X-ray is confirmatory 

for pneumonia. The first troponin is negative. An initial 12 lead EKG reveals abnormal but nonsignificant ST segment changes. There is no cardiologist available and the internist will be the 

admitting physician. 

Scenario 1, Part II 

Mr. John is admitted to a Medical-Surgical (Med-Surg) unit under the care of the internist 

(admitting physician) for observation and treatment. Vital signs including O2 saturation had 

returned to normal. The resident physician informs the attending physician about the condition 

of the patient. The nurse collects the specimens and sends them to the lab in a labeled container. 

The attending physician, during rounds, changes the O2 therapy to “as needed” to maintain 

oxygen saturation above 96%. 

Scenario 1 Part III

Day 2: The nurse, when assessing the patient in the beginning of the shift, notices that the patient 

had a markedly pink itchy irritated right eye with a tearing sensation. The patient reported that 
18

he was on some eye drops in the distant past, but did not recall the name of the medication. The 

doctor who was then providing eye care moved out of town, and he does not remember the 

clinic’s number. 

The nurse had a high suspicion of viral conjunctivitis and puts the patient in contact isolation. 

She asked the physician if he wanted an Ophthalmology consult. Per the physician order, she 

requested the consultant Ophthalmologist. The ophthalmologist, during the consult, was able to 

elicit a history of being in contact with a family member that recently had “pink eyes”. The 

patient was an occasional contact lens wearer but had none recently. The ophthalmologist 

established that the visual acuity, intraocular pressure, and ophthalmic exam were normal except 

for signs and symptoms consistent with viral conjunctivitis. The ophthalmologist ordered topical 

tetracaine 1% to be applied by the ophthalmologist to both eyes (and to have the bottles available 

in the patient’s medication tray for when she rounded on the patient). He ordered that the patient 

be able to self-administer preservative free artificial tears in single-use vials prn (as needed). 

Because of excessive discharge, she also ordered that an assay for adenovirus be available on the 

patient’s floor. The ophthalmologist planned to return the following day or ask the patient to 

come to follow up in the office in 24 hours should the patient be discharged.

Scenario 1, Part IV

Day 3: The troponin levels remained negative, and there was no change in the EKG. Mr. John

had decreased chest pain and cough and other vital signs are normal. The patient stabilizes in 48 

hours with the lab report positive for growth of streptococcus pneumoniae and confirming 

sensitivity to azithromycin. The blood cultures returned negative with no growth. He is 

discharged home with a diagnosis of pneumonia and conjunctivitis to follow up as an outpatient 

in seven days with his primary care physician. He is asked to see the ophthalmologist the next 

day to follow up on his viral conjunctivitis since he left the hospital before seeing him. At 

discharge, the nurse hands over to the patient the summary with the discharge instructions.
19

Figure 1. Sequence of events in first hospital stay included in validation testing
20

Use Case 2: Outpatient care after second hospitalization

Scenario 2, Part I

Mr. John is now a 67-year-old male who presents to the Community General Hospital 

Emergency Department with complaints of headache, and a reported two short periods of 

confusion. He has just recovered from influenza that kept him bedridden at home for 8 days with 

vomiting and diarrhea. Mr. John is a known patient of Type 2 diabetes mellitus and 

hypercholesterolemia. The patient is triaged and taken into the Emergency Department. The ER 

nurse records the vital signs, which are within the normal limits. The Emergency physician 

orders electrolytes, CBC, a chest x-ray, CT scan, EKG, echo cardiogram, carotid artery Doppler 

and calls the internist who admits the patient to the ICU for evaluation of a possible stroke or 

TIA and orders an MRI. The MRI in the hospital is out of order and the MRI is not done. The 

echocardiogram results are scanned into the chart in the media section instead of cardiology 

where they are usually located. All other tests are negative except the lab tests come back with a 

low sodium and potassium and this is resolved within 24 hours by intravenous fluids. He 

receives orders for an MRI outpatient, Coumadin therapy for anticoagulation and teaching by the 

pharmacist.

Mr. John is sent home with a diagnosis of possible transient ischemic attack (TIA) and discharge 

instructions to follow up as an outpatient in seven days with his primary care physician and in 

three days with a neurologist. He receives five prescriptions to fill immediately.

Scenario 2, Part II

Upon coming into the office of his primary care physician on day two after discharge, Mr. John 

reports he feels much better but still gets headaches. The primary care doctor looks for the tests 

done at the hospital. He sees the MRI and echocardiogram were ordered but cannot find the 

results. The physician gets several calls during the outpatient encounter from the emergency 

department where another of his patients, Bill Bates, has arrived with chest pain so he stops and 

takes the urgent call. After addressing the needs of Mr. Bates, lab tests are drawn and Mr. John is 

sent home and will see the neurologist in the morning.

The urgent call is about Mr. Bill Bates. This interruption starts with the nurse in the primary care

physician’s office being called to the phone. The emergency department nurse says that Mr. 

Bates has come to the ED with a severe nosebleed and headache. The emergency department 

needs to know if Mr. Bates is on Coumadin. Mr. Bates cannot remember his medications and left 

them at home.
21

Figure 2. Sequence of events from second hospital day which are reviewed during the outpatient 

visit during validation testing
22

The findings from the data analysis have implications for the design of use cases for ensuring 

safety-enhanced design in the context of these activities, in addition to others:

1. Ordering medications

2. Ordering laboratory tests

3. Ordering diagnostic imaging tests

4. Avoiding drug-drug, drug-allergy, and food-allergy interactions

5. Maintaining the problem list 

6. Maintaining and reconciling the medication list 

7. Maintaining the medication allergy list 

8. Interpreting the recommendations from clinical decision support 

9. Using the electronic medication administration record 

10. Performing clinical information reconciliation and incorporation 

11. Electronic prescribing of medications

When conducting summative usability testing evaluations, representative, appropriately licensed, 

participants simulate the roles of user categories (e.g., eligible professionals and nurses). The 

mapping of the listed activities above and the user category roles are:

1. Eligible professional (Medical Doctor, Nurse Practitioner, or Physician Assistant) role

a. Computerized provider order entry – medications 

b. Computerized provider order entry – laboratory 

c. Computerized provider order entry – diagnostic imaging 

d. Clinical decision support 

e. Drug-drug, drug-allergy interaction checks

f. Problem list

g. Medication allergy list 

h. Medication list 

i. Clinical information reconciliation and incorporation 

2. Nurse role

a. Electronic medication administration record 

b. Vital signs documentation 

c. Clinical decision support 

d. Drug-drug, drug-allergy interaction checks 

e. Medication allergy list 

f. Clinical information reconciliation and incorporation 

g. Retrieve imaging results 
23

Table 3: Usability Test Tasks for Safety-Enhanced Design (Representative Use Cases)

Use Case 1: First Hospital Stay

Scenario 1, Part I: Triage Nurse

Participant is now logged in as the Triage Nurse

Critical Task Scenario Information Usability Test Task

Patient chart has been 

“initiated”

Nurse documents patient’s 

demographics

Patient = Mr. William John

Age = 65 years

Ethnicity = Hispanic

Sex = Male

• Sign on to the System

• Open the patient record for Mr. William John

• Enter the provided demographic information 

for this patient into the System

Nurse enters patient’s 

complaints, current 

illnesses/conditions (history), 

and allergies

Patient complaints = severe 

chest pain, difficulty 

breathing, and productive 

cough

Enter the provided clinical information for this 

patient into his patient record

Current illnesses/conditions 

(history) = Type 2 diabetes 

mellitus, 

Hypercholesterolemia

Allergies = thimerosal and 

“pink sweetener”

Nurse documents triage 

information with disposition 

to Emergency Department 

(ED)

Patient is triaged and taken to 

the ED

Enter information into the patient’s record about 

triaging and transporting him to the ED 
24

Scenario 1, Part I: Emergency Department Nurse

Participant is now logged in as the ED Nurse

Critical Task Scenario Information Usability Test Task

ED nurse verifies 

information documented 

by the Triage nurse

Patient = Mr. William John

Age = 65 years

Ethnicity = Hispanic

Sex = Male

• Sign on to the System

• Open the record for Mr. William John

• Review the provided demographic and clinical 

information that were entered into his record 

by the Triage nurse

Patient complaints = severe 

chest pain, difficulty 

breathing, and productive 

cough

Current illnesses/conditions 

(history) = Type 2 diabetes 

mellitus, 

Hypercholesterolemia

Allergies = thimerosal and 

“pink sweetener”

ED nurse records the 

patient’s vital signs, and 

notifies the ED physician

Temperature, Blood Pressure, 

Respiratory Rate = within 

normal limits for an adult 

male

• Enter the provided clinical information for this 

patient into his record

• Enter a note stating that the ED physician was 

notified about the increased heart rate and low 

O2 Saturation

Heart rate = 110 BPM

O2 Saturation = 90%
25

Scenario 1, Part I: Emergency Physician

Participant is now logged in as the ED Physician

Critical Task Scenario Information Usability Test Task

ED physician verifies 

information documented 

by the Triage and ED 

nurses

Patient = Mr. William John

Age = 65 years

Ethnicity = Hispanic

Sex = Male

• Sign on to the System

• Open the record for Mr. William John

• Review the provided demographic and clinical 

information that were entered into the record 

for this patient by the Triage and ED nurses 

Patient complaints = severe 

chest pain, difficulty 

breathing, and productive 

cough

Current illnesses/conditions 

(history) = Type 2 diabetes 

mellitus, 

hypercholesterolemia

Allergies = thimerosal and 

“pink sweetener”

Temperature, Blood Pressure, 

Respiratory Rate = within 

normal limits for an adult 

male

Heart rate = 110 BPM

O2 Saturation = 90%

ED physician orders O2 

therapy, 12-lead EKG, lab 

tests, a radiology test, 

telemetry, medications, 

and diabetic diet

Orders = 

• O2 therapy per nasal 

cannula STAT

• 12-lead EKG

• Serial troponin levels

• Sputum for gram stain, now

• Complete Blood Count 

(CBC)

• Finger-stick glucose

• Chest X-ray

• Continuous telemetry 

monitoring

• Long-acting insulin

• Regular insulin on slidingscale

• Atorvastatin

Enter the provided orders for this patient into his 

record
26

Critical Task Scenario Information Usability Test Task

• Vitamins

• Aspirin

• Azithromycin

• 1800-calorie diabetic diet

Scenario 1, Part I: Emergency Department Nurse

Participant is now logged in as the ED Nurse

Critical Task Scenario Information Usability Test Task

ED nurse documents 

medication administration, 

O2 therapy, EKG 

performed, lab specimens 

obtained, glucose test 

performed, and telemetry 

monitoring

Treatments & tests 

performed =

• Continuous O2 therapy 

per nasal cannula initiated

• Continuous telemetry 

monitoring initiated

• 12-lead EKG completed

• Finger-stick glucose 

completed

Lab specimens collected =

• Serial troponin level

• Sputum for gram stain

• Complete Blood Count 

(CBC)

Medications administered = 

• Regular insulin on 

sliding-scale

• Atorvastatin

• Vitamins

• Aspirin

• Azithromycin

• Sign on to the System

• Open the record for Mr. William John

• Enter the provided information for medication 

administration, O2 therapy initiated, EKG 

completed, finger-stick glucose completed (and 

result), and specimens collected for this patient 

into his record
27

Scenario 1, Part I: Emergency Physician

Participant is now logged in as the ED Physician

Critical Task Scenario Information Usability Test Task

ED physician reviews 

results of diagnostic tests 

Diagnostic Test Results = 

• Chest X-ray confirms 

pneumonia

• Troponin is negative

• EKG shows abnormal, 

but non-significant, ST 

segment changes

• Sign on to the System

• Open the record for Mr. William John

• Find and review the provided diagnostic test 

results 

ED physician orders the 

patient to be admitted as an 

inpatient

Admit patient to MedicalSurgical unit today for 

observation and treatment

Enter admission order into the patient’s record

Scenario 1, Part II: Inpatient Nurse

Participant is now logged in as the Inpatient Nurse

Critical Task Scenario Information Usability Test Task

Inpatient nurse receives 

the patient on the 

Medical-Surgical unit and 

verifies information 

documented by the Triage 

nurse, ED nurse, and ED 

physician, and reviews 

diagnostic test results and 

the order for inpatient 

admit 

Patient = Mr. William John

Age = 65 years

Ethnicity = Hispanic

Sex = Male

• Sign on to the System

• Open the record for Mr. William John

• Find and verify the documented demographic 

information

• Find and verify the documented patient 

complaints, current illnesses/conditions, 

allergy, vital sign, and O2 saturation 

information

• Find and review the documented medication 

administration information

• Find and review the information documented 

for the treatments and tests performed 

• Find and review the diagnostic test results 

• Find and verify the inpatient admit orders

Patient complaints = severe 

chest pain, difficulty 

breathing, and productive 

cough

Current illnesses/conditions 

(history) = Type 2 diabetes 

mellitus, 

hypercholesterolemia

Allergies = thimerosal and 

“pink sweetener”

Temperature, Blood Pressure, 

Respiratory Rate = within 

normal limits for an adult 

male
28

Critical Task Scenario Information Usability Test Task

Heart rate = 110 BPM

O2 Saturation = 90%

Treatments & tests performed 

=

• Continuous O2 therapy per 

nasal cannula initiated

• Continuous telemetry 

monitoring initiated

• 12-lead EKG completed

• Finger-stick glucose 

completed

Medications administered = 

• Regular insulin on slidingscale

• Atorvastatin

• Vitamins

• Aspirin

• Azithromycin

Diagnostic Test Results = 

• Chest X-ray confirms 

pneumonia

• Troponin is negative

• EKG shows abnormal, but 

non-significant, ST 

segment changes

Admission order = Admit 

patient to Medical-Surgical 

unit today for observation and 

treatment

Inpatient nurse documents 

vital signs information 

and O2 Saturation 

information

O2 Saturation = 98% Enter the provided clinical information for this 

patient into his record

Temperature, Heart Rate, 

Blood Pressure, Respiratory 

Rate information = within 

normal limits for an adult 

male
29

Scenario 1, Part II: Attending Physician

Participant is now logged in as the Attending Physician

Critical Task Scenario Information Usability Test Task

Attending physician makes 

rounds and orders the O2 

Therapy changed to as 

needed

Order = change O2 therapy 

per nasal cannula from 

“continuous” to “prn, to 

maintain O2 Saturation > 

96%”

• Sign on to the System

• Open the record for Mr. William John

• Enter the provided O2 therapy change order into 

this patient’s record

Scenario 1, Part III: Inpatient Nurse

Participant is now logged in as the Inpatient Nurse

Critical Task Scenario Information Usability Test Task

On Day 2, the inpatient 

nurse assesses the patient at 

the beginning of the shift, 

and observes that his right 

eye is markedly pink, itchy, 

irritated, and tearing; the 

nurse documents these 

findings

Suspecting viral 

conjunctivitis, the nurse 

puts the patient in contact 

isolation and documents this 

action

The nurse reports the 

findings to the attending 

physician and asks if she 

wants an Ophthalmology 

consult; based on the 

physician’s verbal order, the 

nurse orders the consult

Patient’s right eye is 

markedly pink, itchy, 

irritated, and tearing

• Sign on to the System

• Open the record for Mr. William John

• Enter the provided information about the 

patient’s eye into his record

• Enter the provided information about the patient 

being placed on contact isolation into his record

• Enter into the patient’s record the provided 

information about the condition of the patient’s 

eye being reported to the physician

• Enter into the patient’s record the provided 

information for verbal order on behalf of the 

attending physician for an Ophthalmology 

consult

Patient has been placed on 

contact isolation

Condition of patient’s eye 

reported to attending 

physician
30

Critical Task Scenario Information Usability Test Task

Ophthalmology consult for 

possible viral conjunctivitis 

ASAP

Scenario 1, Part III: Ophthalmologist

Participant is now logged in as the Ophthalmologist

Critical Task Scenario Information Usability Test Task

The ophthalmologist 

performs a consult and 

documents her findings and 

plan

• Patient has a history of 

being in contact with a 

family member who 

recently had “pink eye”

• Patient was an occasional 

contact lens wearer, but 

has not been wearing 

them recently

• Visual acuity, intraocular 

pressure, and ophthalmic 

exam are normal except 

for signs and symptoms 

consistent with viral 

conjunctivitis

• Plan: will return 

tomorrow or ask patient to 

schedule a follow-up visit 

in 24 hours if patient is 

discharged before rounds 

tomorrow

• Sign on to the System

• Open the record for Mr. William John

• Enter the provided findings and Plan into this 

patient’s record

The ophthalmologist orders 

an eye medication

• Topical tetracaine 1%, to 

be administered in both 

eyes by ophthalmologist 

(have bottles available in 

patient’s medication tray 

for physician’s rounds

Enter the provided order for the eye medication 

into the patient’s record

The ophthalmologist orders 

eye drops for the patient to 

administer to himself

• Preservative-free artificial 

tears in single-use vials 

for patient selfadministration

Enter the provided order for the artificial tears 

into the patient’s record

The ophthalmologist orders 

an assay for adenovirus

• Keep an assay for 

adenovirus available on 

the patient’s hospital unit

Enter the provided order for the adenovirus assay 

into the patient’s record
31

Scenario 1, Part IV: Attending Physician

Participant is now logged in as the Attending Physician

Critical Task Scenario Information Usability Test Task

Attending physician reviews 

diagnostic test results, vital 

signs, clinical observations

• Troponin levels 

remained negative

• No change in the EKG

• Decreased chest pain and 

cough

• Vital signs are normal

• Patient is stable 

• Sputum culture and 

susceptibility report is 

positive for growth of 

streptococcus 

pneumoniae with 

sensitivity to 

azithromycin

• Blood cultures were 

negative/no growth

• Sign on to the System

• Open the record for Mr. William John

• Find and review the provided clinical 

information that is documented in the patient 

record, including diagnostic test results, vital 

signs, and other assessments

Attending physician 

documents a discharge 

summary 

The troponin levels have 

remained negative, and there 

has been no change in the 

EKG. Mr. John has 

decreased chest pain and 

cough, and other vital signs 

are normal. The patient has 

stabilized over the past 48 

hours. The lab report for 

sputum specimen is positive 

for growth of streptococcus 

pneumonia, with 

confirmation of sensitivity to 

azithromycin. The blood 

cultures returned negative 

with no growth. Patient to be 

discharged home with a 

diagnosis of pneumonia and 

conjunctivitis.

Enter the provided discharge summary 

information into the patient’s record

Attending physician orders 

the patient to be discharged 

from the hospital to home 

with a final diagnosis of 

pneumonia and 

conjunctivitis, and with 

instructions to schedule a 

visit with his primary care 

physician and a 

Discharge patient to home

Final diagnosis pneumonia 

and conjunctivitis

Patient to schedule a visit 

with his primary care 

physician in seven days

Enter the provided discharge order information 

into the patient’s record
32

Critical Task Scenario Information Usability Test Task

recommendation to schedule 

a visit with the 

ophthalmologist the next 

day for follow up on his 

viral conjunctivitis 

Recommended that patient 

schedule a visit with the 

ophthalmologist for 

tomorrow for follow up on 

the viral conjunctivitis
33

Use Case 2: Outpatient Care Testing (inpatient data is only to prepopulate hospital chart not 

to test)

Scenario 2, Part I: Triage Nurse

Participant is now logged in as the Triage Nurse, 

Critical Task Scenario Information Usability Test Task

Patient chart has been 

“initiated”

Nurse documents patient’s 

demographics

Patient = Mr. William John

Age = 67 years

Ethnicity = Hispanic

Sex = Male

• Sign on to the System

• Open the patient record for Mr. William John

• Enter/verify the provided demographic 

information for this patient into the System

Nurse enters patient’s 

complaints, current 

illnesses/conditions (history), 

and allergies

Patient complaints = 

headache and reported two 

short periods of confusion

Enter the provided clinical information for this 

patient into his patient record

Current illnesses/conditions 

(history) = Type 2 diabetes 

mellitus, 

hypercholesterolemia

Nurse documents triage 

information with disposition 

to Emergency Department 

(ED)

Patient is triaged and taken to 

the ED

Enter information into the patient’s record about 

triaging and transporting him to the ED 

Scenario 2, Part I: Emergency Department Nurse

Participant is now logged in as the ED Nurse

Critical Task Scenario Information Usability Test Task

ED nurse records the 

patient’s vital signs

Temperature, Heart Rate, 

Blood Pressure, Respiratory 

Rate = within normal limits 

for an adult male

• Sign on to the System

• Open the record for Mr. William John

• Enter the provided clinical information for this 

patient into his record
34

Scenario 2, Part I: Emergency Physician

Participant is now logged in as the ED Physician

Critical Task Scenario Information Usability Test Task

ED physician orders lab 

tests, a radiology test, 

EKG, and other diagnostics 

tests

Orders = 

• Electrolytes

• Complete Blood Count 

(CBC)

• Chest X-ray

• CT Scan of head

• EKG

• ECHO cardiogram

• Carotid artery Doppler

• Sign on to the System

• Open the record for Mr. William John

• Enter the provided orders for this patient into 

his record

Scenario 2, Part I: Internist

Participant is now logged in as the Internist

Critical Task Scenario Information Usability Test Task

Internist orders admit to 

ICU and additional 

diagnostics testing

Orders = 

• Admit to ICU for 

evaluation of possible 

stroke or TIA

• MRI

• Sign on to the System

• Open the record for Mr. William John

• Enter the provided orders for this patient into 

his record

Internist reviews results of 

diagnostic testing

• Electrolytes results = low 

Na+ and K+

• Complete Blood Count 

(CBC) result = normal

• Chest X-ray = negative

• CT Scan of head = 

negative

• EKG = negative

• ECHO cardiogram = 

misfiled in media section 

of patient record

• Carotid artery Doppler = 

negative

Review the provided results in the patient record

Internist orders outpatient 

diagnostic testing, 

medication with teaching, 

and discharge with followup

Orders = 

• MRI outpatient

• Coumadin with teaching 

by pharmacist

• Discharge to home, final 

diagnosis possible TIA

• Patient to schedule a visit 

with his primary care 

physician in seven days

Enter the provided discharge orders for this patient 

into his record
35

Critical Task Scenario Information Usability Test Task

• Patient to schedule a visit

with neurologist in three 

days

Scenario 2, Part II: Primary Care Physician 

Participant is now logged in as the Primary Care Physician

Critical Task Scenario Information Usability Test Task

Primary care physician 

looks for diagnostic results 

from tests performed in the 

hospital 

• MRI and ECHO 

cardiogram results not 

available

• Electrolytes results = low 

Na+ and K+

• Complete Blood Count 

(CBC) result = normal

• Chest X-ray = negative

• CT Scan of head = 

negative

• EKG = negative

• ECHO cardiogram = 

misfiled in media section 

of patient record

• Carotid artery Doppler = 

negative

• Sign on to the System

• Open the record for Mr. William John

• Search for MRI and ECHO cardiogram results 

in the patient’s record

• Review the provided results that are available in 

the patient’s record

Primary care physician is 

interrupted by call from 

Emergency department 

regarding Mr. Bates who 

has presented with a nose 

bleed. 

Physician moves to Mr. 

Bates chart. Physician opens 

Mr. Bates chart and reviews 

his history and medication. 

• Physician opens Mr. Bates chart for review

• Physician closes Mr. Bates chart and is able to 

go back to Mr. William Johns chart.
36

6 Empirically Based Human Factors Guidance for SafetyEnhanced Design of Health Information Technology

Human factors guidance for safe and usable design exists for a number of safety-critical 

domains, but not currently for health information technology (HIT) design. For example, 

the Human Factors Design Standard (HFDS) is a compilation of human factors principles 

and guidance for the procurement, design, development, and testing of aviation systems, 

facilities, and equipment.22 Within this document, the principle of ‘simplicity’ is defined 

as “Information should be presented simply and in a well-organized manner.” In order to 

achieve this principle, the following guidance is provided, along with other 

recommendations: "Information should be presented in consistent, predictable locations.” 

Based upon our empirical findings, detailed in the previous section, as well as a 

comprehensive review of existing standards under NIST Contract Number 

SB134110CN0107, we provide a limited set of high-priority guidance below. These 

guidelines for standardization are targeted uniquely at eliminating ‘never events’ and 

associated patient harm by proactively addressing and mitigating the root causes of 

critical use errors from EHR design and implementation elements, as distinguished in our 

framework on the relationship between usability and patient safety, as described in Figure 

3 of NISTIR 7804 Technical Evaluation, Testing and Validation of the Usability of 

Electronic Health Records.

23

The empirically based human factors guidance for safety-enhanced design is:

1. Consistently display information critical to patient identification in a reserved area (specified 

below) to avoid wrong patient errors

1.1 Patient identification information shall be displayed in the upper left hand corner of 

all screens/windows in a consistent order; so that users can efficiently and accurately 

find and verify patient identity

1.2 The information shall continue to be displayed in the same location regardless of 

scrolling or other navigational mechanisms to move within the screen/window

1.3 The order shall be to first display the patient’s name with the last (family) name 

capitalized, followed by a comma and then first (given) name, middle name, and 

modifier, followed by date of birth using e.g., Nov 9, 1961 format and age and 

gender, and then followed by MRN number.

1.4 For mobile devices or tablets with smaller screen sizes, it may be preferable to display 

the information horizontally using the same ordering convention and white space 

between the three elements. The information should be demarcated on the bottom
37

and/or the side, such as by employing white space, shading, or a line, from additional 

optional identifiers

1.5 An example of this reserved area is:

SMITH, Walter Joseph III

Nov 9, 1961 (53 yo M)

MRN1348887 

 

a. NAME: The last (family) name should be first and capitalized followed by a ‘,’ and 

space prior to a capitalized first (given) name with the rest of the name in lower 

case. The capitalization is used to distinguish the last name in cases of ambiguity 

(e.g., Clark Kelly could be Clark KELLY or Kelly CLARK). It also reduces variation for 

names with multiple capitalizations, such as McDonald.

b. NAME MODIFIER: In the absence of a modifier (e.g., Jr, Sr, III), nothing shall be 

displayed in that location

c. DATE: The month represented as the first three letters of the month (or four in 

languages other than English such as Italian where this is needed to 

disambiguate months) shall be represented with a capitalized first letter with the 

rest in lower case in order to make the capitalized last name more distinguishable 

quickly on the display. The full year shall be displayed as four numeric digits

d. AGE: Displaying the age reduces the cognitive work required by the user to 

convert date of birth into age. For years old, the display convention is ‘yo’ with a 

space after the number, rounded down to the nearest digit. Similarly, months old 

is displayed as ‘mo’, weeks old as ‘wo’, and days old as ‘do’. In Neonatal Intensive 

Care Units, DOL 1 is often used for the first day of life, which corresponds to 0 

days old. Similarly, DOL 2 is the second day of life. Decisions on when to display 

yo, mo, wo, do, and DOL are expected to vary by institution. For example, a 

hospital may display DOL for the first five days of life, followed by do until 30 

days old, then wo until 24 weeks old, then mo until 24 months old, and finally yo 

after 24 months of age. For the purposes of tracking accuracy of information, it 

should be possible to display ‘on demand’ the value of the age in the original 

format in which it was stored or transferred with interoperable systems. Age for 

patients should not be displayed in values of less than 1 unit (e.g., 0.0001 yo)

e. GENDER: For gender, the display options should be M or Male for Male, F or 

Female for Female, and Other. Additional details specifying subcategories under 

Other, as necessary, shall be viewable on demand, such as transgender, or 

reasons for a gender change

f. MRN: The allocation of digits to the Medical Record Number (MRN) should be 

able to be modified in the future to accommodate future changes. Additional 

identifiers such as care episode can be included on this line after the MRN. The 
38

font size for MRN and other numeric identifiers can be smaller than the other 

information displayed in the reserved area or placed to the right of the name and 

date of birth information, but should still be viewable by older users.

4 MRN 

information may be displayed in the reserved area only in response to an explicit 

user action and/or when a barcoded wristband is scanned. Other identifiers, such 

as encounter numbers, shall not be displayed in the reserved area in order to 

reduce the likelihood of confusing the identifiers

g. ADDITIONAL IDENTIFIERS: Optional additional identifiers shall not be included in 

the reserved area, as defined by being below a clearly demarcated horizontal line 

or to the right of the area above the demarcation line. The display of optional 

identifiers should not cover task-critical information except for short periods ‘on 

demand.’ Additional optional identifiers include:

1) Place of birth

2) Picture. Note: Recommended to be a color picture of an individual patient 

taken within the last 5 years, with no other individuals in the picture, and as a 

close-up of the head facing the camera.

3) Biometrics

4) Genome

5) Barcode

6) Episode/encounter code

7) Suspected, confirmed or ruled out to have a highly infectious disease (e.g., 

‘Confirmed Ebola’), etc.

 4 Kochurova, Olga, Joan K. Portello, and Mark Rosenfield. "Is the 3× reading rule appropriate 

for computer users?." Displays 38 (2015): 38-43.
39

2. Provide visual cues to reduce risks of entering information and writing orders in the wrong 

patient’s chart

2.1 Visually differentiate a chart5 that enables a user to have unrestricted access to input 

information (i.e., input mode) from a chart, which restricts the user’s ability to input 

information (i.e., view-only mode)

2.2 Enable user to enter information on only one patient’s chart at one time

2.3 Enable user to have a chart in view-only mode in parallel with a chart with unrestricted 

access to input information in order to support specialty-specific care needs (e.g., 

coordinated mother and child care following a birth, coordinated care of multiple birth 

patients)

2.4 Enable user to easily transition from the current chart with unrestricted access to input 

information to another chart by a deliberate action (i.e., identification/activation of the

patient chart), by the user.

2.4.1 Categories of charts that are likely to be needed by clinical providers are 

1) charts for patients that are scheduled to be seen in the near future (e.g., 

24 hours), 2) charts for patients that have recently had information input 

into them, 3) charts that have ordered laboratory tests or imaging tests 

that are pending results, and 4) charts that have planned actions such as 

documenting progress notes which have not yet been completed. 

2.4.2 Easily transition implies that context has to be preserved in a way that is 

clear to the user when the user transitions to another chart and back to a 

previous chart; and that it is easy to find and identify a desired patient's 

chart for any relevant patient in the system. Context should be preserved 

in these transitions such that unsaved work in progress text should be 

preserved by the system until saved (or deleted) by the user

2.5 Visually distinguish the mechanism for moving within a single patient’s chart and 

transitioning from one chart with unrestricted access to input information to another

 5 In the 2002 article "Maintaining a Legally Sound Health Record." Journal of AHIMA 73(2), a 

chart is defined as "generated at or for a healthcare organization as its business record and is the 

record that would be released upon request. It does not affect the discoverability of other 

information held by the organization. The custodian of the legal health record is the health 

information manager in collaboration with information technology personnel. HIM professionals 

oversee the operational functions related to collecting, protecting, and archiving the legal health 

record, while information technology staff manage the technical infrastructure of the electronic 

health record."
40

3. Support efficient and easy identification of inaccurate, outdated, or inappropriate items in lists 

of grouped information by having information presented simply and in a well-organized 

manner.

Ways to achieve this include the following:

3.1 Lists of patients assigned to a particular clinician user should be presented in consistent, 

predictable locations within and across displays and print-outs and the content should 

not vary based on display location.

3.2 The status of a note and order as draft as compared to final shall be clearly indicated on 

appropriate displays.

3.3 Clearly indicate the method by which the system saves information, whether auto-save or 

requiring deliberate action to save, or combinations thereof. 

3.4 Inputted information should be automatically saved when a user transitions from one 

chart to another.

3.5 The language used should be task-oriented and familiar to users, including being 

consistent with expectations based upon clinical training.

3.6 Enable a user to easily order medications that have a high likelihood of being the 

appropriate medication, dose, and route. The likelihood is increased when displays are 

tailored to specialty-specific user requirements, comply with national evidence-based 

recommendations, are in accordance with system, organizational, unit, or individual 

provider preferences specified in advance, or are similar to orders made by the same 

physician on similar patients, on the same patient in the past, or providers with similar 

characteristics.

3.7 Support assessing relationships of displayed information and allowing users with 

appropriate permissions to modify locations and relationships for inaccurately placed 

information, including laboratory results, imaging results, pathology results, consult 

notes, and progress notes. This includes information within a single patient’s chart as well 

as information placed in the wrong patient’s chart. The information about the time and 

person that made the change should be viewable on demand.
41

7 Conclusions

This report examines the use of EHRs and their potential to improve and hinder healthcare

quality and patient safety. We drew upon five different methods of data collection, utilized 

cross-cutting analytic methods, and had analysts from different disciplines and perspectives. The 

results demonstrate strong congruence among the data, across the methods, and amongst the 

analysts. Three major critical use risk areas surfaced during analysis: 1) Identification of 

information; 2) Consistency of information; and 3) Integrity of information. These three areas

are consistent with findings from other studies that examined the use of EHRs. 

Ultimately, the data from this study demonstrate that during safety-critical tasks and times,

patient safety is negatively affected, in part because mistakes and critical use errors occur more 

frequently and because users are highly frustrated, and thus more likely to employ workarounds, 

such as relying upon supplemental artifacts, e.g., paper ‘shadow charts’ or whiteboards. Figure 3

below highlights the ways in which problems with identification, consistency, and integrity of 

information can result in suboptimal and unsafe patient care.

Figure 3. Relationship between usability findings and safe and effective clinical care
42

In accordance with the empirical evidence, and in order to make care more optimal and safe, we 

provided human factors guidance for improving the usability of EHRs. There are three areas for 

this guidance: 

1) Consistently displaying information critical to patient identification in a reserved area to 

avoid wrong patient errors, 

2) Providing cues to reduce risks of entering information and writing orders in the wrong 

patient’s chart, and 

3) Supporting efficient and easy identification of inaccurate, outdated, or inappropriate 

items in lists of grouped information by having information presented simply and in a 

well-organized manner. 

Finally, we provided two use cases for use during summative usability testing to achieve the goal 

of safety-enhanced design by validating that potential patient safety risks are proactively 

addressed and/or mitigated, one for the inpatient setting and one for the outpatient setting. These 

use cases may also prove useful for organizations that tailor aspects of the design during 

implementation and optimization processes to protect against inadvertently introducing new risks 

to patients.
43

8 References

1 Buntin, M., Burke, M., Hoaglin, M., & Blumenthal, D. (2011). The benefits of health 

information technology: a review of the recent literature shows predominantly positive results. 

Health Affairs, 30 (3), 464-471.

2 Gibbons, M.C., et al. (2009). Impact of consumer health informatics applications. Evidence 

Report/Technology Assessment, Oct (188), 1-546; and Cebul, R., Love, T., Jain, A.K., & Hebert, 

C.J. (2011). Electronic health records and quality of diabetes care. New England Journal of 

Medicine, 365, 825-833.

3 Longhurst, C., Parast, L., Sandborg, C., et al. (2010). Decrease in hospital-wide mortality rate 

after implementation of a commercially sold computerized physician order entry system. 

Pediatrics, 126, 14-21.

4 Institute of Medicine. (2001). Crossing the quality chasm: a new health system for the twentyfirst century. Washington, DC: National Academy Press.

5 Classen, D.C., et al. (2011). Global trigger tool shows that adverse events in hospitals may be 

ten times greater than previously measured. Health Affairs, 3(4), 581-589.

6 Han, Y.Y., Carcillo, J., Venkataraman, S., et al. (2005). Unexpected increased mortality after 

implementation of a commercially sold computerized physician order entry system. Pediatrics, 

116(6), 1506 –1512. Correction in Pediatrics, 117(2), 594. 7 Horsky J, Kuperman GJ, Patel VL. Comprehensive analysis of a medication dosing error 

related to CPOE. J Am Med Inform Assoc 2005; 12(4):377-82.

8 Hruschka, D. J., Schwartz, D., Cobb St. John, D., Picone-Decaro, E., Jenkins, R. A., Carey, J.

W. (2004). Reliability in coding open-ended data: Lessons learned from HIV behavioral

research. Field Methods, 16, 307-331.

9 Erlandson, D. A., Harris, E. L., Skipper, B. L., & Allen, S. D. (1993). Doing naturalistic 

inquiry. Thousand Oaks, CA: Sage.

10 Lincoln, Y. S., & Guba, E. G. (1985). Naturalistic inquiry. Thousand Oaks, CA: Sage. 11 Charmaz, K. (2006). Constructing grounded theory: A practical guide through qualitative

analysis. Thousand Oaks, CA: Sage. 12 Saldaña, J. (2013). The coding manual for qualitative researchers. Thousand Oaks, CA: 

Sage.

13 Saldaña, J. (2013). The coding manual for qualitative researchers. Thousand Oaks, CA: 

Sage.

14 Richards, L., & Morse, J. M. (2007). User’s guide for qualitative methods (2nd ed). Thousand 

Oaks, CA: Sage.

15 Charmaz, K. (2006). Constructing grounded theory: A practical guide through qualitative

analysis. Thousand Oaks, CA: Sage.

16 Saldaña, J. (2013). The coding manual for qualitative researchers. Thousand Oaks, CA: 

Sage.

17 Saldaña, J. (2013). The coding manual for qualitative researchers. Thousand Oaks, CA: 

Sage.

18 Saldaña, J. (2013). The coding manual for qualitative researchers. Thousand Oaks, CA: 

Sage.

19 Lincoln, Y. S., & Guba, E. G. (1985). Naturalistic inquiry. Thousand Oaks, CA: Sage.

20 Patton, MQ. (1999). Enhancing the quality and credibility of qualitative analysis. HSR: Health 

Services Research, 3, 1189-1208.

 
44

 21 Mays, N. & Pope, C. (2000). Qualtative research in health care: Assessing quality in 

qualitative research. BMJ: British Medical Journal, 320, 50-52.

22 Ahlstrom, V., & Longo, K. (2003). Human Factors Design Standard (HF-STD-001). Atlantic 

City International Airport, NJ: Federal Aviation Administration William J. Hughes Technical 

Center.

23 Svetlana, L.Z., Matthew, Q.T., Ramaiah, M., Schumacher, R.M., Patterson, E.S., North, R., 

Zhang, J., Gibbons, M.C., Abbott, P. “NISTIR 7804 Technical Evaluation, Testing and 

Validation of the Usability of Electronic Health Records.” NIST Interagency/Internal Report 

(NISTIR) 7804, 2012. Available at: http://www.nist.gov/customcf/get_pdf.cfm?pub_id=909701.