Assessing the Risk of Wrong-Patient Errors in an Electronic Health Record That Allows Multiple Records Open
Phase: NA
Conditions: Medical Errors, Medical Order Entry Systems
Sponsor: Columbia University

1 
 
FINAL PROTOCOL 
Title: Assessing the Risk of Wrong-Patient Errors in an EMR that Allows Multiple Records Open 
Principal Investigator: Jason Adelman, MD, MS 
Sponsor: AHRQ Agency for Healthcare Research and Quality 
Grant ID: R21HS023704 
Date: 6/21/2017 
 
BACKGROUND AND SIGNIFICANCE 
Frequency of wrong-patient orders. At least 70,000 U.S. physicians use computerized provider 
order entry (CPOE) to place orders.1,2 Although CPOE is associated with a reduction in medical errors,3-7 
when orders are placed electronically certain types of errors, including placing orders on the wrong 
patient, may occur more frequently.8-13 
 
The danger of wrong-patient electronic orders was highlighted by one hospital’s report that after 
implementing CPOE, medications were prescribed for the wrong patient several times per month.14 In 
2003, the United States Pharmacopeia analyzed 7,029 voluntarily reported medication errors over a 7-
month period and found a mean of 9 wrong-patient orders at each of 120 participating institutions using 
CPOE.15 This report likely under-estimated the extent of wrong-patient electronic orders, as voluntary 
reporting is known to be an unreliable method for identifying errors.16-17 Dr. Jason Adelman, the Principal 
Investigator of this proposal, developed an automated surveillance tool, the Wrong-Patient Retract-and 
Reorder measure, that identified 5,246 orders placed on the wrong patient in one year at a single 
academic medical center, with a rate of 58 wrong-patient orders per 100,000 orders.18 In that year, 1 in 6 
providers placed an order on the wrong-patient, and 1 in 37 hospitalized patients had an order placed for 
them in error. This was the first study using automated surveillance to identify wrong-patient orders, and 
it demonstrated the prevalence of wrong-patient orders to be significantly higher than previously thought. 
 
Potential risk of placing an order on the wrong patient when multiple medical records are 
open at once. Although there have been no studies quantifying (or even establishing) an increased risk 
of wrong-patient errors when providers have multiple records open at once, there have been several 
articles and expert opinions that warn of this potential risk. In an abstract presented at the 2012 
American Medical Informatics Association (AMIA) conference, researchers presented a survey of Chief 
Medical Information Officers (CMIOs) evaluating the causes of wrong-patient errors in CPOE systems, 
and reported “[CMIO respondents] attributed wrong-patient errors to the ability to view multiple charts on 
one computer simultaneously and poor screen design.”19 In a 2012 study, Hyman et al hypothesized that 
having multiple records open at once increased the rate of wrong-patient errors, but reported they were 
“unable to determine whether there was any link between the reported errors and multiple records having 
been open.”20 
 
A 2013 study published in JAMIA evaluated 32 wrong-patient errors, and noted that 60% of these 
errors occurred in systems that allowed at least two charts open simultaneously. Investigators did not 
measure total orders, and therefore could not determine error rates and quantify the relationship between 
the number of records open at a time and the risk of wrong-patient errors.21 A 2013 white paper titled, 
How to Identify and Address Unsafe Conditions Associated with Health Information Technology 
published by the Office of the National Coordinator for Health Information Technology (ONC) claimed 
that data can be entered “incorrectly into the electronic record due to multiple records being open,” but 
no reference was provided supporting this statement.22 
 

2 
 
The mechanism by which multiple patient records open simultaneously can lead to a wrong-
patient error may be related to the ease with which users can toggle between patient records and the 
similar looking computer screens. The magnitude of this risk needs to be established to help IT 
leadership decide on how to safely implement CPOE systems. There have been no studies that have 
evaluated whether multiple records open increase the risk of wrong-patient errors, by how much, and if 
any increase is dependent on the number of records open (i.e., is four records open simultaneously more 
dangerous than three? Is three worse than two?). Our research is an important first step in quantifying 
this risk. 
 
Practice of allowing multiple records opens at once. Investigators on this protocol conducted 
a national survey of Chief Medical Informatics Officers (CMIOs) via the American Medical Informatics 
Association listserv and the Association of Medical Directors of Information Systems listserv in March 
2014.23 The survey assessed the practice of allowing multiple records open at once, as well as CMIOs’ 
experiences and perceptions of the risks and gains in efficiencies of having multiple records open. There 
was a wide range of configuration of this feature across facilities. Of 167 facilities with CPOE systems 
capable of simultaneously displaying multiple patient records at once, 44.3% allowed three or more 
records open (unrestricted environment), and 38.3% limited the system to only one record open at a time 
(restricted environment). It is interesting to note that a number of respondents changed their settings 
some time after their initial configuration: three hospitals decreased the number of records allowed open 
at once; two hospitals increased the number of records allowed open at once; and one hospital went 
from allowing four records open at once down to one, and then back up to two.Table 1 provides some 
comments from CMIOs explaining why they chose to establish an unrestricted environment or restricted 
environment, and why some hospitals decided to switch configurations after the initial instillation. The 
results of this survey demonstrate that IT leadership is lacking objective data to guide their decision on 
how many patient records allowed open at once in EHRs. 
 
Table 1. CMIO considerations in configuring an restricted vs unrestricted EHR environment. 
Examples of comments from CMIOs who established a restricted environment. 
 My organization chose to allow only one EHR open at a time…. We feel that multiple records open and in 
use by the same person is not good practice and is an error waiting to happen. 
 Our software vendor allows three records to be opened at once. We made the decision to only allow one 
record to be opened. We also analyzed our facility in terms of noise levels and distractions and decided that 
having multiple charts open had the potential of a significant patient safety issue. 
Examples of comments from CMIOs who established an unrestricted environment. 
 The need to multitask is inherit in today’s practice of medicine. We are commonly called to provide 
coverage for patients for a simple task. To leave a chart and then return becomes a high burden. 
 I think the efficiency benefits are such that this is justified. There are other ways to prevent wrong-patient 
problems. 
Examples of comments from CMIOs who changed configurations after the initial installation. 
 We had wrong-patient errors when we let users access up to three charts at one time, which is why we now 
limit to just one. 
 Our system for many years only allowed one patient chart to be open. Due to some complaints and 
arguments that it would increase efficiency, a decision was made to allow more than one chart to be 
opened. Less than a year later, it got reverted back because of the increase in documentation errors as 
reported by the HIMS department. 

3 
 
The feasibility of this study is a result of developing a validated and reliable measure of 
wrong-patient errors. Adelman et al developed 
the Wrong-Patient Retract-and-Reorder (RAR) 
measure as an automated method for identifying 
wrong-patient electronic orders.18 This measure 
will be used to identify the primary outcome in 
this study. It works by identifying orders placed 
for a patient that are retracted within 10 minutes 
and then placed by the same provider for a 
different patient within the next 10 minutes 
(Figure 1). We performed real-time confirmatory 
telephone interviews with providers who placed 
and retracted orders to validate the RAR 
measure. These phone interviews with ordering 
providers demonstrated that the RAR measure 
correctly identified near-miss errors in 170 of 
223 cases (positive predictive value 76.2%). 
 
Near-miss errors in patient safety research. The Wrong-Patient RAR measure identifies near-
miss errors rather than errors that reach the patient and cause harm. Near-miss errors are also referred 
to as “close calls” by the Department of Veterans Affairs,24 “good catches” by the National Association for 
Healthcare Quality,25 and “free lessons” by the safety expert James Reason.26 The use of near-miss 
errors to test safety improvements in healthcare is encouraged by every major patient safety organization 
including the Agency for Healthcare Research and Quality (AHRQ), Institute of Medicine (IOM), World 
Health Organization (WHO), Institute for Healthcare Improvement (IHI), and The Joint Commission 
(JCAHO) because they have been shown by safety experts to have the same causal pathway as errors 
that cause harm.27-31 
 
The link in the causal pathways is demonstrated graphically in the Incident Causation Model first 
described by industrial safety expert T.W. Van der Schaaf (Figure 2).27 In this model, the key distinction 
between an adverse event and a near-miss 
error is that in the latter a “human recovery” 
occurs, just before the error reaches a patient 
and causes harm. This principle is the 
foundation for the RAR measure, which 
identifies self-caught errors. In a study by Bates 
et al of 4,031 randomly selected patient 
records, 247 adverse drug events and 194 
near-miss drug events had similar underlying 
causes.32 Because near-miss and actual errors 
have similar proximate causes, interventions 
that reduce near-miss errors should also reduce 
actual errors. In fact, in the seminal article that 
first demonstrated that CPOE systems prevent 
medication errors, Bates et al demonstrated 
that CPOE systems decreased both serious 
errors and near-miss errors.4 
AHRQ’s position on the use of 
Figure 2. Incident Causation Model. 
Figure 1. Wrong-Patient Retract-and-Reorder Measure.  

4 
 
near-miss errors. AHRQ has supported the use of near-miss errors in patient safety initiatives. The 
AHRQ National Resource Center for Health Information Technology developed the Health Information 
Technology Evaluation Toolkit, and lists near-miss errors as a particularly useful outcome measure for 
the evaluation of the effectiveness and safety of Health Information Technology (HIT) projects.33 Since 
2005, AHRQ has coordinated the development and maintenance of the “Common Formats” for 
national reporting of patient safety events to Patient Safety Organizations (PSOs), and has included 
near-miss errors as important patient safety data to be collected and analyzed.34 In addition, AHRQ 
currently has 141 articles under the “Near-Miss Collection” on the AHRQ Patient Safety Network 
website, which reviews research and lessons learned specifically related to near-miss errors.35 
 
Harm from wrong-patient orders. Until recently, most knowledge about harm from wrong-
patient orders came from anecdotal reports. For example, in March 2011 the Institute for Safe 
Medication Practices (ISMP) published a case report of a physician who used CPOE and accidentally 
ordered a sedative and paralytic agent for the wrong patient, resulting in respiratory arrest and 
death.36 Prior to the Montefiore study, there were no publications that attempted to quantify harm or 
potential harm from wrong-patient orders. To examine the potential for harm associated with wrong-
patient orders, we examined 5,246 near-miss errors identified using the RAR tool and classified 971 
errors as clinically significant, 264 as serious, and 126 as life threatening.18 
 
Preliminary Studies 
Determining the rate of wrong-patient orders. Adelman and colleagues developed and 
validated the Wrong-Patient RAR measure as an automated tool for capturing wrong-patient orders. After 
validating the Wrong-Patient RAR measure, we applied it to the complete data set of over 9 million 
electronic orders placed at a large academic medical center in one year.18 The Wrong-Patient RAR 
measure found that 1,388 providers placed 6,885 orders that were retracted and reordered. Multiplying 
the proportion of retracted orders identified by the positive predictive value of 76.2%, we estimated that 
5,246 wrong-patient orders were placed, with an average of 14 retracted orders per day, 1 in 6 providers 
placed an order on the wrong-patient and retracted it, and 1 in 37 hospitalized patients had an order 
placed for them in error. 
 
Using the Wrong-Patient RAR measure in a randomized controlled trial evaluating two 
interventions for preventing wrong-patient orders. We conducted a three-arm randomized controlled 
trial to test two interventions designed to prevent wrong-patient orders: an ID-verify alert that displayed 
the patient’s name, gender, and age for the provider to verify with one click; and an ID-reentry function 
that blocked access to the order entry screen until the provider entered the patient’s initials, gender, and 
age.18 Over 4,000 providers who placed inpatient orders were randomly assigned to receive the ID-verify 
alert, the ID-reentry function, or neither. Over one million orders were placed in each arm of the study. 
Compared with the control condition, the ID-verify alert reduced the odds of a wrong-patient order by 
16% and the ID-reentry function reduced the odds of a wrong-patient order by 41%. The study found that 
wrong-patient orders are common, and that interventions like the ID-verify alert and the ID-reentry 
function can lower the frequency of wrong-patient orders. Although a 41% reduction of wrong-patient 
errors is substantial, additional safety measures are needed to achieve more complete protection for 
patients.  
 
 
 
 
 

5 
 
SPECIFIC AIMS 
 
Aim 1. Compare the rate of wrong-patient orders in a restricted environment that limits providers to 
one patient record open at a time in an electronic health record to an unrestricted environment that 
allows providers to open a maximum of four records at once. 
 
Aim 2. Conduct subgroup analyses to examine the rate of wrong-patient orders stratified by clinical 
location, including emergency department, outpatient, and inpatient settings.  
 
STUDY DESIGN 
Two-arm Randomized Trial. In a randomized comparative effectiveness trial, we propose to randomly 
assign providers to one of two configurations of the electronic health record (EHR) system: 1) a restricted 
mode that limits providers to open one record at a time; or 2) an unrestricted mode that allows providers 
to open a maximum of four records open at once. The trial will compare the rate of wrong-patient orders 
between trial arms, identified by the Wrong-Patient Retract-and-Reorder (RAR) measure. 
 
Role of Columbia University Medical Center (CUMC). CUMC will act as the lead institution and have 
oversight over study implementation and data analysis. No data will be collected for CUMC patients. 
Final de-identified data sets will be transferred using a secure file transfer protocol to the principal 
investigator at CUMC from the study biostatistician at Albert Einstein College of Medicine. 
 
Study Sites. The trial will be conducted at Albert Einstein College of Medicine/Montefiore Medical 
Center, an integrated regional health system and academic medical center in New York. Study sites 
include four hospitals, five emergency departments, and more than 100 ambulatory facilities. The health 
system will use the EpicCare EHR system during this study; Epic has committed to support the project in 
implementing the Wrong-Patient RAR measure as well as developing and programming a log to capture 
the number of records open at the time of placing each order. Montefiore met the following six criteria for 
inclusion as a study site: (1) the capability to implement the Wrong-Patient RAR measure and the log to 
record the number of records open at the time of placing each order; (2) the capability to merge the 
results of the Wrong-Patient RAR measure with a table indicating patient, provider, and order 
characteristics; (3) a minimum of 
30,000 admissions per year; (4) a 
minimum of 50,000 emergency 
department visits per year; (5) a 
minimum of 500,000 outpatient 
visits per year; and (6) patients 
representing low-income groups, 
minority groups, the elderly, and 
individuals with special health 
care needs (Table 2). 
 
 
 
METHODS 
 
Orders versus Order 
Sessions. If a provider begins placing orders in the wrong patient’s record, there is the possibility that 
several such orders will be placed and then retracted together. Therefore, individual orders do not 
Table 2. Characteristics of Study Site. 
Systems 
 
Location 
Bronx, NY 
Inpatient beds 
1536 
Annual admissions 
90,000 
Emergency department visits 
306,000 
Outpatient visits 
2,000,000 
Priority Populations 
 
% Minority 
72% 
% Medicaid/uninsured 
60% 
% Medicare 
11% 

6 
 
represent independent opportunities for errors to occur. Orders are clustered within order sessions, 
defined as a series of orders placed consecutively by a single provider for a single patient that begins 
with opening that patient’s order file and terminates when an order is placed on another patient or after 
60 minutes, whichever comes first. Thus the order session, rather than each order, represents an 
independent opportunity for a wrong-patient error to occur. 
 
Unit of Analysis. The unit of analysis will be the order session.  
 
Primary Outcome. The primary outcome measure, the Wrong-Patient RAR measure, is an 
electronic query run retrospectively against every order placed during the study period to identify 
instances in which one or more orders placed for a patient were retracted (cancelled) by the same 
provider within 10 minutes, and then reordered by the same provider for a different patient within the next 
10 minutes (RAR events). In the validation study, phone interviews with ordering providers confirmed that 
the RAR measure correctly identified wrong-patient orders in 170 of 223 events, yielding a positive 
predictive value of 76.2% (95% confidence interval, 70.6% to 81.9%).18 The primary outcome is wrong-
patient order sessions, defined as order sessions that include a wrong-patient RAR event identified by 
the Wrong-Patient RAR measure.  
 
Provider Level, Patient Level, Order-Session, and Order Level Covariates. The data will 
have a nested, hierarchical structure with order sessions clustered within providers. The analysis will 
account for this hierarchical structure and we will gather from the electronic medical record attributes of 
the provider, patient, order session, and order. 
Provider-level covariates: type of ordering provider (attending, resident, physician assistant, nurse 
practitioner, pharmacist, or other), and total number of orders placed during the study period (a measure 
of the frequency with which the provider uses the system). 
Patient-level covariates: age, race, ethnicity, sex, unit, and date and time of admission.  
Order-session level covariates: location of the order session (emergency department, medical-
surgical unit, intensive care unit, labor and delivery, pediatrics, other specialty units).  
Order level covariates: type of order (medication, imaging, nursing order, procedure, other), date 
and time of order, date and time of retraction, and number of patient records open at the time the order 
was placed. 
 
Inclusion and Exclusion Criteria 
Patient Inclusion and Exclusion Criteria. All patients are at risk for wrong-patient orders, so all 
orders placed for all patients will be included in the study. We will ask to waive informed consent to 
patient inclusion in this study, as it poses no more than minimal risk to patients. To enhance 
confidentiality, we will replace medical record numbers with pseudo-identifiers in the analytic data sets 
prior to analysis. 
 
Provider Inclusion and Exclusion Criteria. Any provider who can place an electronic order can 
potentially place an order on the wrong patient. Prior work showed that wrong-patient errors are made by 
physicians (60 errors per 100,000 orders), nurse practitioners and physician assistants (74 errors per 
100,000 orders), nurses (33 errors per 100,000 orders) and pharmacists (67 errors per 100,000 
orders).18 We will therefore include in the study all providers with the authority to place electronic orders. 
Providers will be excluded only if their workflow either 1) has a defined requirement to open two patient 

7 
 
records simultaneously (eg, mother-infant services), or 2) bypasses the standard order entry process and 
therefore would not be captured by the outcome measure (eg, radiologists). 
 
To protect providers’ identities, all provider identifiers will be replaced with pseudo-identifiers in 
the analytic data sets prior to analysis. We will request a waiver of consent from the Institutional Review 
Board (IRB) as this study poses no more than minimal risk for providers (see Protection of Human 
Subjects).  
 
Randomization 
All inpatient, emergency department, and outpatient providers will be randomized in a 1:1 ratio to 
either a maximum of one patient record open at a time (restricted mode), or a maximum of four records 
open at once (unrestricted mode). Assignments will be made prior to the start of the study, and will 
remain constant throughout the study. A computer programmer working in the information technology 
(IT) department, who is not an investigator on this study, will use a computerized random number 
generator and assign one random number to each provider. Providers assigned odd numbers will be 
assigned to the restricted arm, and those assigned even numbers will be assigned to the unrestricted 
arm. Providers will be manually assigned per randomization to EHR user-role templates that differ only in 
the number of patient records allowed open. Providers newly hired after the start of the study will be 
assigned a random number when assigned a new user logon for the EHR from a computer programmer 
not affiliated with the study, and will be added to the appropriate arm based on their assigned random 
number. The study methodology has been approved by the Senior Vice President and Chief Medical 
Officer at Montefiore, as well as the Executive Lead for the Epic EHR.  
 
Data Capture 
The data required to define orders, orders sessions, and the number of open patient records, 
patient and provider characteristics, and RAR events is automatically captured by the Epic EHR system. 
Investigators will collaborate with information technology personnel at the study sites and with Epic 
throughout the implementation phase of the study to develop the methodology for extracting the required 
data. At the end of the data collection period, data for all orders placed by study providers during the 
study period will be extracted, de-identified, encrypted, and transmitted electronically to the investigators 
for analysis. Preset batch orders (eg, for vaccines) will be excluded from the analysis, as these orders 
are not under the control of individual providers. These data sets will provide all of the information 
needed to carry out the analyses. 
To examine the rate of wrong-patient orders and proportion of orders placed when one, two, 
three, or four records were open, an electronic log will be developed and programmed into the Epic 
system to record the number of records open at the time each order was placed.  
 
Protection of Human Subjects 
Confidentiality and Consent. Study data will be extracted retrospectively at the midpoint and at 
the end of the study period. All data sets, reports, and other study records will be de-identified for 
analysis. Patient medical record numbers and provider ID numbers will be replaced with pseudo-
identifiers, and other personal identifiable information (PII) will be deleted from the data sets. The 
randomized trial will examine two EHR configurations that are both in wide use, with neither representing 
an established best practice. The randomized trial also does not involve any procedures for which written 
consent is normally required, and also does not present more than minimal risk to either patients or 
providers. All near-miss errors will be evaluated in the aggregate and no PII will be used in presentations, 
publications, or reports. Information will not be released except as necessary for monitoring by the IRB. 

8 
 
As such, per the Code of Federal Regulations Title 45, Parts 160 and 164a, we seek an exemption to the 
requirement for use of the HIPAA Authorization form, and have completed the HIPAA Authorization 
Exemption request form. Per the code of Federal Regulations Title 45, Part 46.116 (d), we ask the IRB to 
waive informed consent, and have completed the Informed Consent Waiver request form. 
 
Data Security and Integrity. All files will be kept on password-protected HIPAA-compliant 
computers in locked offices. Only will research personnel, namely the computer programmer, 
biostatistician, and prinicipal investigator, will have access to the study data. Analytic data sets will be de-
identified, with patient and provider identifiers replaced with pseudo-identifiers. Data files will be imported 
into the native format of the statistical program to be used for analysis. The data will be checked for 
internal consistency. Identified anomalies will be examined, and corrections to the data will be made and 
documented as necessary. 
 
Communication with providers. At the start of the randomized trial, investigators will explain the 
purpose of the study to clinical staff via email and directly from within the IT system, using a message 
crafted by the study team. The message will assure providers that data will be kept confidential and 
cooperation will carry no risk to them. 
 
Unintended consequences. All unintended consequences and confidentially breaches detected 
will be brought to the attention of the Chief Medical Information Officer, the Chief Medical Officer, and 
overseeing IRB. 
  
Study Limitations 
First, the Wrong-Patient RAR measure is designed to identify near-miss errors and does not 
capture wrong-patient errors that reach patients. However, near-miss errors follow the same causal 
pathways as errors that reach the patient and cause harm, occur more frequently, and can be reliably 
measured without the biases inherent in voluntary self-reports by providers. National and international 
patient safety and regulatory agencies endorse the use of near-miss errors to evaluate safety 
interventions.27-31 Second, orders that are retracted and reordered beyond the 10-minute timeframe (10 
minutes to retraction, 10 minutes to reorder) will not be identified by the RAR measure. However, prior 
work demonstrated that the average retract-and-reorder time was less than 2 minutes; the 10-minute 
timeframe will detect the majority of retracted near-miss order errors and identify a sufficient number of 
outcome events to power this study. Finally, this research is limited to only one EHR vendor system. 
However, when fully rolled out, Epic’s clients will provide care for 45%-55% of the U.S. population.  
 
 
 

9 
 
References  
1. 
United States Department of Labor. United States Bureau of Labor Statistics. Occupational outlook 
handbook, 11th edn. 2010. (http://www.bls.gov/oco/pdf/ocos074.pdf). 
2. 
Davis MW. The State of US Hospitals Relative to Achieving Meaningful Use Measurements. 
HIMSS Analytics. 2009. (http://www.himssanalytics.org/docs/HA_ARRA_100509.pdf). 
3. 
Bates DW, Leape LL, Cullen DJ, et al. Effect of computerized physician order entry and a team 
intervention on prevention of serious medication errors. JAMA 1998;280:1311-1316. 
4. 
Bates DW, Teich JM, Lee J, et al. The impact of computerized physician order entry on medication 
error prevention. J Am Med Inform Assoc 1999;6:313-321. 
5. 
Kaushal R, Shojania KG, Bates DW. Effects of computerized physician order entry and clinical 
decision support systems on medication safety: a systematic review. Arch Intern Med 
2003;163:1409-1416. 
6. 
Reckmann MH, Westbrook JI, Koh Y, Lo C, Day RO. Does computerized provider order entry 
reduce prescribing errors for hospital inpatients? A systematic review. J Am Med Inform Assoc 
2009;16:613-623. 
7. 
Radley DC, Wasserman MR, Olsho LE, et al. Reduction in medication errors in hospitals due to 
adoption of computerized provider order entry systems. J Am Med Inform Assoc 2013;20:470-476. 
8. 
Koppel R, Metlay JP, Cohen A, et al. Role of computerized physician order entry systems in 
facilitating medication errors. JAMA 2005;293:1197-1203. 
9. 
Harrison MI, Kopel R, Bar-Lev S. Unintended consequences of information technologies in health 
care: an interactive sociotechnical analysis. J Am Med Inform Assoc 2007;14:542-9. 
10. Ash JS, Sittig DF, Poon EG, Guappone K, Campbell E, Dykstra RH. The extent and importance of 
unintended consequences related to computerized provider order entry. J Am Med Inform Assoc 
2007;14:415-423. 
11. Campbell EM, Sittig DF, Ash JS, Guappone KP, Dykstra RH. Types of unintended consequences 
related to computerized provider order entry. J Am Med Inform Assoc 2006;13:547-555. 
12. Sittig DF, Ash JS. Clinical Information Systems: Overcoming Adverse Consequences. Boston, MA: 
Jones and Bartlett, 2010. 
13. Ash J, Berg M, Coiera E. Some unintended consequences of information technology in health care: 
the nature of patient care information system-related errors. J Am Med Inform Assoc 2003;11:104-
112. 
14. Cohen M. Medication Errors, 2nd ed. Washington DC: American Pharmaceutical Association, 
2006. 
15. United States Pharmacopeia. MEDMARX 5th Anniversary Data Report: A Chartbook of 2003 
Findings and Trends 1999-2003. Rockville, MD: United States Pharmacopeia, 2004. 
16. Classen DC, Resar R, Griffin F, et al. ‘Global trigger tool’ shows that adverse events in hospitals 
may be ten times greater than previously measured. Health Aff (Millwood) 2011;30:581-589. 
17. Levinson D. Adverse Events in Hospitals: Methods for Identifying Events. Washington, DC: United 
States Department of Health and Human Services, Office of Inspector General, 2010. 
18. Adelman JS, Kalkut GE, Schechter CB, et al. Understanding and preventing wrong-patient 
electronic orders: a randomized controlled trial. J Am Med Inform Assoc 2013;20:305-310. 
19. Levin HI, Levin JE, Docimo SG. “I meant that med for Baylee not Bailey!": a mixed method study to 
identify incidence and risk factors for CPOE patient misidentification. AMIA Annu Symp Proc 
2012:1294-1301.  
20. Hyman D, Laire M, Redmond D, Kaplan DW. The use of patient pictures and verification screens to 
reduce computerized provider order entry errors. Pediatrics 2012;130:211-219. 

10 
 
21. Galanter W, Falck S, Burns M, Laragh M, Lambert BL. Indication-based prescribing prevents 
wrong-patient medication errors in computerized provider order entry (CPOE). J Am Med Inform 
Assoc 2013;20:477-481.  
22. The Office of the National Coordinator for Health Information Technology. Anticipating Unintended 
Consequences of Health Information Technology and Health Information Exchange: How to Identify 
and Address Unsafe Conditions Associated with Health IT. Washington, DC, 2013. 
23. Adelman JS, Berger MA, Rai A, et al. A national survey assessing the number of records allowed 
open in electronic health records at hospitals and ambulatory sites. J Am Med Inform Assoc 
2017;24:992-995. 
24. Department of Veterans Affairs. VHA National Patient Safety Improvement Handbook, 2011. 
(http://www.va.gov/vhapublications/ViewPublication.asp?pub_ID=2389). 
25. National Association for Healthcare Quality. National Association for Healthcare Quality (NAHQ) 
Issues 'Call to Action' for Enhancing Health Care Quality and Patient Safety. 2012. 
(http://www.prnewswire.com/news-releases/national-association-for-healthcare-quality-nahq-
issues-call-to-action-for-enhancing-health-care-quality-and-patient-safety-174423331.html). 
26. Reason, J. Human error: models and management. BMJ: British Medical Journal 2000;320:768.  
27. Institute of Medicine (IOM). Patient Safety: Achieving a New Standard for Care. Washington, DC: 
National Academy Press, 2004. 
28. Agency for Healthcare Research and Quality: Doing What Counts for Patient Safety: Federal 
Actions to Reduce Medical Errors and Their Impact Report of the Quality Interagency Coordination 
Task Force (QuIC) To the President, 2000 (http://archive.ahrq.gov/quic/Report/errors6.pdf). 
29. Leape L, Abookire S. WHO Draft Guidelines for Adverse Event Reporting and Learning Systems: 
From Information to Action. Geneva, Switzerland: World Health Organization, 2005. 
(http://www.who.int/patientsafety/events/05/Reporting_Guidelines.pdf). 
30.  Institute for Healthcare Improvement. "Create a Reporting System” April 2011. 
(http://www.ihi.org/knowledge/Pages/Changes/CreateaReportingSystem.aspx). 
31. Wu, Albert W. The Value of Close Calls in Improving Patient Safety: Learning How to Avoid and 
Mitigate Patient Harm. Oakbrook Terrace, IL: Joint Commission Resources, 2011. 
32. Bates DW, Cullen DJ, Laird N, et al, Incidence of adverse drug events and potential adverse drug 
events. JAMA 1995;274:29-34. 
33. Cusack CM, Poon EG. Health Information Technology Evaluation Toolkit. Prepared for the AHRQ 
National Resource Center for Health Information Technology under contract No. 290-04- 0016. 
AHRQ Publication No. 08-0005-EF. Rockville, MD: Agency for Healthcare Research and Quality. 
November 2007. 
34. Agency for Healthcare Research and Quality. Concepts Underlying Common Formats Event 
Descriptions. (http://www.pso.ahrq.gov/formats/eventdesc.htm. 
35. AHRQ Patient Safety Network. The Near Miss Collection. 
(http://psnet.ahrq.gov/collectionBrowse.aspx?taxonomyID=508). 
36. Institute for Safe Medication Practices. Oops, sorry, wrong patient! A patient verification process is 
needed everywhere, not just at the bedside. 2011. 
(http://www.ismp.org/Newsletters/acutecare/articles/20110310.asp). 
 
 

11 
 
FINAL STATISTICAL ANALYSIS PLAN 
Descriptive Analysis. We will use descriptive analysis to examine provider-level, patient-level, 
order session-level, and order-level covariates for the orders placed in the restricted versus unrestricted 
mode. Descriptive statistics will be reported as mean and standard deviation for continuous variables, 
and as count and percentages for categories. 
Orders versus Order Sessions. If a provider begins placing orders in the wrong patient’s record, 
there is the possibility that several such orders will be placed and then retracted together. Therefore, 
individual orders do not represent independent opportunities for errors to occur. Orders are clustered 
within order sessions, defined as a series of orders placed consecutively by a single provider for a single 
patient that begins with opening that patient’s order file and terminates when an order is placed on 
another patient or after 60 minutes, whichever comes first. Thus the order session, rather than each 
order, represents an independent opportunity for a wrong-patient error to occur.  
 
Primary Outcome. The primary outcome measure, the Wrong-Patient RAR measure, is an 
electronic query run retrospectively against every order placed during the study period to identify 
instances in which one or more orders placed for a patient were retracted (cancelled) by the same 
provider within 10 minutes, and then reordered by the same provider for a different patient within the next 
10 minutes (RAR events). The primary outcome is wrong-patient order sessions, defined as order 
sessions that include a wrong-patient RAR event identified by the Wrong-Patient RAR measure.  
 
Primary Analysis (Aim 1). The primary analysis of the randomized trial is intention-to-treat, with 
each provider generating a cluster of order sessions. Hence, the order session will be used as the unit of 
analysis. The primary outcome variable is dichotomous, indicating whether or not each order session 
contains a wrong-patient RAR event. To determine the effect of trial arm on wrong-patient orders, we will 
construct a random effects logistic regression model with wrong-patient order sessions as the outcome, 
and randomization arm as the independent variable, using provider as a random intercept to account for 
clusters of order sessions within providers. We will estimate the effect of the restricted mode versus the 
unrestricted mode on the rate of RAR events using the odds ratio and its 95% confidence interval (CI), 
and will test the null hypothesis using the Wald test with a two-tailed significance level of 0.05. The 
primary outcome is reported as the number of wrong-patient order sessions per 100,000 order sessions. 
 
Subgroup Analysis (Aim 2). Additional analyses of the effect of the restricted mode versus the 
unrestricted mode in specified clinical locations will be carried out by including indicators for the locations 
and location-study arm interaction terms in the model. For subgroup analyses, we will compare the rate 
of RAR events in emergency department, outpatient, and inpatient settings, and more specifically in 
inpatient units, including medical/surgical, critical care, pediatrics, and obstetrics units. We will construct 
similar mixed-effects logistic regression models for each predefined subgroup, with a separate model 
including an interaction term to test the significance of treatment effects across subgroups, using the 
Wald test for significance. 
 
As-Treated Analysis. Because of administrative errors, some providers were not assigned to the 
trial arm to which they were randomized. Therefore, we will repeat all assessments in as-treated 
analyses (ie, according to treatment received) such that each order or order session was characterized 
by the provider’s configuration at the time the orders were placed. 
 
For orders placed in the unrestricted mode, we will examine the rate of RAR events and the 
percentage of orders placed when one, two, three, or four records were open at the time of ordering, 

12 
 
overall and stratified by clinical setting. For these analyses the order will be used as the unit of analysis 
(rather than the order session), because a provider may open or close patient records while placing a 
series of orders during a single order session. We will report the number of RAR events per 100,000 
orders along with 95% binomial confidence intervals, as well as the percentage of all orders placed under 
those conditions. These results will be presented for the unrestricted arm overall, and disaggregated by 
clinical location. 
 
Sample Size. For pragmatic reasons, sample size is not determined by the investigators but will 
represent the data that is available during the study period. Moreover, as noted, orders do not represent 
independent events, but are clustered within order sessions that, in turn, are clustered within providers. 
The magnitude of the effects at these levels are not currently known. Based on prior experience in this 
setting, in 1 year of data collection we expected to obtain approximately 9,000,000 orders in 2,400,000 
order sessions, placed by 3,200 distinct providers for 440,000 distinct patients. We assumed 100 wrong-
patient order sessions per 100,000 order sessions in the Unrestricted arm, and an intra-provider 
correlation of 0.01. With accrual of approximately 1,600 providers in each randomization arm, and a trial 
duration yielding an average of approximately 1,300 order sessions per provider with a coefficient of 
variation of 1.2, the trial would have >99% power to detect an odds ratio of 0.5 (or lower), 96% power to 
detect an odds ratio of 0.6, and 76% power to detect an odds ratio of 0.7. 
 
Missing Data. Due to the automatic functioning of the electronic health record, we do not 
anticipate that there will be missing data concerning RAR events, the system configuration of restricted 
mode vs. unrestricted mode, the number of records open at the time of placing an order, or provider-level 
covariates. There may be sporadic missing information regarding the patient-level covariates. In 
preliminary data, in a data set of more than 11.6 million orders, there were no missing observations for 
patient age, sex, or race. An indicator for Hispanic ethnicity was missing in just 4 records. With similar 
levels of missing data, we feel comfortable that restricting regression models to only include cases with 
complete data will not introduce appreciable levels of bias. If we find that any variables are missing for 
more than 1 record per 1,000 (after backfilling based on other records involving the same patient), we will 
extend our analyses to address this. We will presume that data are missing at random and will apply 
multiple imputation with chained equations. 
 
Interim Analysis. To safeguard against the possibility that our intervention actually worsens 
(increases) the rate of RAR events, and to prevent unnecessary continuation of a study that is already 
conclusive, we will have a data safety monitoring committee conduct one interim review of the data in the 
randomized trial. After 6 months, we expect 50% of the data to have been accrued. Using the Lan-
Demets alpha spending procedure with symmetric O’Brien-Fleming boundaries, the stopping rule at the 
interim review will be a z-statistic of magnitude 3.0318 or greater.1 The associated nominal p-value is 
0.0024. Combined with a final analysis using a critical z-value of 1.9669 (nominal P=0.0492), we will 
have spent our overall alpha of 0.05 at the end of the study. The effects of this interim analysis procedure 
on nominal statistical power (see above) is negligible, less than 0.5 percentage points, so no adjustments 
to data collection need be made to account for this. 
 
Reference 
1. Demets DL, Lan KK. Interim analysis: the alpha spending function approach. Stat Med. 
1994;13:15e30.