The Use of Non-Interruptive Alerts for Improving the Use of Clinical Decision Rules in the Emergency Department: A Cluster Randomized Controlled Trial
Phase: NA
Conditions: Pulmonary Embolism, Deep Vein Thrombosis, Minor Head Injury, Chest Pain, Neck Trauma
Sponsor: University of Utah

 
 
 
 
 
 
 
 
 
The Use of Non-Interruptive Alerts for Improving the Use of 
Clinical Decision Rules in the Emergency Department: A Cluster 
Randomized Controlled Trial 
 
 
 
Principal Investigator: 
Keaton Morgan, MD1,2,3 
 
Other Investigators: 
Kensaku Kawamoto, MD, PhD, MHS1,2 
Devin Horton, MD2,4 
Polina V. Kukhareva, PhD, MPH, MS1,2 
Troy Madsen, MD3 
Phillip B. Warner, MS1,2 
 
Institutional affiliations: 
1 – Department of Biomedical Informatics, University of Utah, Salt Lake City, Utah, USA 
2 – University of Utah Health, Salt Lake City, Utah, USA 
3 - Department of Surgery, University of Utah, Salt Lake City, Utah, USA 
4 – Department of Internal Medicine, University of Utah, Salt Lake City, Utah, USA 
 
 
November 17th, 2020 
 
 
 
 
 
 
 

 
I. BACKGROUND AND SIGNIFICANCE 
The increased use of computed tomography (CT) imaging studies in the acute care setting has 
been a growing concern in recent decades, with overall use of CT increasing 330% from 1996 to 
2007.1 This trend has been identified as a key area of improvement in delivering high quality, 
low cost care and is an area of active research and public health focus.2 One suggested method 
for addressing the issue is the use of validated clinical decision rules that risk stratify patients and 
provide evidence-based guidance on diagnostic imaging for a variety of conditions in the acute 
care setting. The American College of Emergency Physicians (ACEP), for example, has 
advocated the use of several of these rules in best practice clinical guidelines3 and their Choosing 
Wisely recommendations.2 
Despite these efforts, the adoption rates of clinical decision rules in acute care is low. Buchanan 
et al.,4 for example, performed a study at the University of Utah from 2010-2015 that 
demonstrated that over 25% of patients had diagnostic testing for pulmonary embolism even 
though it was not recommended by clinical decision rules. Similar results have been found at 
other institutions,5,6 with an estimated 33% of CT scans performed for suspected pulmonary 
being potentially avoidable, if clinical decision rules were universally adopted.6 
There is some evidence that clinical decision support (CDS) can be used to improve the 
utilization rates of clinical decision rules,7–9 but results from these studies are mixed and there is 
a great need for more high quality studies on the subject.10–12 Wang et al.,13 for example, 
performed a systematic review and meta-analysis on the impact of pulmonary embolism decision 
rules on CT use and yield. Out of eight total studies included, there was only one randomized 
controlled trial and only one of the seven observation studies was rated as high quality. Overall, 
the authors concluded that implementation of the Wells’ Criteria for Pulmonary Embolism 
through CDS was associated with a modest increase in CT angiography diagnostic yield but 
there was “a lack of cluster-randomization trials to confirm efficacy of clinical decision rules for 
pulmonary embolism.”13 
There is also a need for studying novel CDS delivery methods for the recommendation of 
decision rules. An overwhelming majority of the published studies utilize pop-up alerts,7 forced 
data entry at order placement13 or a combination of the two.11 There is evidence, however, that 
these CDS delivery methods may not be optimal. Westafer et al.,14 for example, performed a 
qualitative study looking at provider reported barriers to applying clinical decision rules for 
pulmonary embolism. Three important barriers they identified were 1) lack of knowledge of rule 
application 2) timing of the CDS (i.e. too late in the workflow) and 3) alert fatigue. There is a 
need for implementing and studying CDS methods that address these barriers. 
In this study, we plan to evaluate the impact of non-interruptive alerts on the use of clinical 
decision rules in the emergency department. Providers will be randomly split into an intervention 
and control groups. The intervention will be patient-specific, non-interruptive alerts suggesting 
the use of one or more clinical decision rules plus listing the suggested clinical decision rules on 
the default screen of an EHR-integrated medical reference tool (MDCalc Connect). The alerts 
will be Epic Storyboard Best Practice Advisory alerts and will include a link for launching the 
MDCalc Connect app from the alert. 

 
This intervention represents a novel CDS approach that effectively addresses the three 
aforementioned barriers to clinical decision rule application (lack of knowledge, CDS timing and 
alert fatigue). In addition, this study will add a randomized controlled trial to an important area 
of research currently lacking high quality clinical trials. 
 
II. STUDY OBJECTIVE(S); INCLUDING SPECIFIC AIMS AND/OR HYPOTHESES 
This study is an extension of a planned quality improvement project that aims to promote 
standard of care by increasing the use of evidence-based clinical decision rules amongst 
emergency medicine providers in the University of Utah Emergency Department. Patient-
specific information from the EHR will be used to recommend the use of relevant clinical 
decision rules to emergency medicine providers at the point-of-care. These recommendations 
will be in the form of non-interruptive alerts with one-click access to the suggested decision rules 
through the MDCalc Connect EHR add-on application. 
To evaluate the efficacy of this quality improvement project, we will perform a cluster 
randomized controlled trial by splitting participating emergency medicine providers into 
intervention and control groups. A cluster randomized controlled trial (RCT) study design is 
necessary for two reasons: 1) there is a great need for RCTs on the study topic (as described in 
the Background and Significance section) and 2) three clinical decision rules included in the 
study are related to venous thromboembolic (VTE) disease, and it has been shown that COVID-
19 infection increases the rates of VTE. Given the unpredictable and evolving nature of the 
COVID-19 pandemic, a before-after study design is a poor choice since the rates of VTE will 
likely be changing over time. 
The specific aims are to determine 1) if the patient-specific non-interruptive alerts increase the 
use of clinical decision rules amongst emergency medicine providers and 2) if an increase in the 
use of clinical decision rules affects provider ordering habits. The ordering habits of interest will 
be tests relevant to the specific clinical decision rules included in the study and will be described 
in detail in the Data Analysis section of this protocol. 
 
III. METHODS 
A. Study Design 
This study will be a single center, non-blinded, cluster randomized controlled trial. The study 
site will be the emergency department of a Level 1 trauma center and tertiary referral 
hospital. It has an annual patient volume of ~60,000 patient visits per year. All resident 
physicians and advanced practice providers practicing in the emergency department during 
the study period will be eligible for participation. Eligible participants will be split roughly 
evenly between intervention and control groups through a stratified random sampling 
technique. The intervention will be two-part: 1) a non-interruptive alert in the EHR 
recommending the use of one or more clinical decision rules and 2) a new default tab in the 
MDCalc Connect app that displays the same clinical decision rules recommended in the non-
interruptive alert. Secondary analysis will include retrospective data up to one year prior to 
study initiation. This historical data will be used as an additional control.  

 
 
B. Study Population 
Provider Participation for RCT: The system will be implemented in the University of Utah 
Emergency Department. Up to 125 providers could be eligible for trial participation and will 
include all resident physicians and advanced practice clinicians (APCs) who practice in the role 
of emergency medicine provider at the study site during the study period.  Providers will be 
split roughly evenly into intervention and control groups as part of a cluster randomized 
controlled trial. Only providers in the intervention group will receive the recommended decision 
rule alerts. 
Provider participation for retrospective chart review: The same criteria as listed in the RCT 
description will be used to identify providers for retrospective chart review. This data will be 
used as a historical control and may include data up to 1 year prior to the RCT study start date. 
Up to 125 additional providers could meet inclusion criteria. 
Patient participation for RCT: All patients evaluated in the University of Utah Emergency 
Department during the study period with a qualifying chief complaint and age 16 years or old 
will be eligible for participation in the trial. Up to 10,000 patients could be eligible for 
participation. The list of qualifying chief complaints was curated by a panel of clinical experts 
who have extensive experience with the decision rules included in the study. They selected chief 
complaints based off of likelihood of the chief complaint being relevant to the included clinical 
decision rules. For example, the Canadian CT Head Rule (which is used for patients with minor 
head trauma), includes chief complaints such as "fall", "head injury" and "motor vehicle 
collision."  
Patient participation for retrospective chart review: The same criteria as described for the RCT 
will be used for patients in the retrospective chart review. This data will be used as a historical 
control and may include data up to 1 year prior to the RCT study start date. Up to 10,000 
additional patients could be included in this portion of the study. 
C. Assessment of Resources 
This study is supported by clinical leadership, and the clinical decision rules selected for 
inclusion in the study are supported by hospital protocols, clinical leadership or both. We have 
estimated that a study period of four months will be the minimum duration required to reach 
statistical power of 0.80 or greater for the study’s primary outcome (clinical decision rule usage).  
 
D. Study Procedures 
Cluster randomized trial: All eligible providers will be split into intervention and control 
groups prior to trial initiation using a randomized sampling technique. At study initiation, all 
providers in the intervention group will receive the clinical decision rule alerts + calculator 
suggestions in the MDCalc Connect app for all patients that meet inclusion criteria. Providers in 
the control group will not receive the alerts.  
Intervention description: The intervention will contain two components: 1) a non-interruptive 
best practice advisory (BPA) in the Epic Storyboard and 2) a default suggested calculator tab in 
the MDCalc Connect app. Six clinical decision rules will be included: Canadian CT Head Rule, 

 
Canadian C-spine Rule, HEART Score, PERC Rule, Wells' PE Criteria and Wells' DVT Criteria. 
Predefined criteria based on chief complaint, patient age and vital signs will be used to determine 
when a decision rule may be relevant in a patient encounter. When a patient meets all of the 
predefined criteria, the intervention will be triggered. 
The non-interruptive alert will display in the Epic Storyboard column and will informs the user 
that a decision rule may be relevant to this patient encounter. For example, it may read "The 
PERC Rule may be relevant to this patient encounter." A hyperlink to the MDCalc application 
will be included in the alert. If the provider opens the MDCalc application, the recommended 
decision rule(s) will be displayed on the app home page. The function of the MDCalc app will 
otherwise be unaffected and the provider may use the app in its usual way.  
The decision rules selected are supported by the American College of Emergency Physicians and 
the study site medical director leadership team and are considered standard of care. Use of the 
decision rules is expected to improve the quality of patient care and diagnostic yield of imaging 
studies and decrease the radiation exposure and cost of care for patients. The intervention alerts 
are not expected to impact any providers at the study site other than the providers participating in 
the study.  
 
IV. DATA COLLECTION 
Following the trial, data for analysis will be extracted from the University of Utah data 
warehouse, system logs and MDCalc Connnect application logs. This will include data for the 
RCT and the retrospective chart review, which will include identical inclusion criteria and 
outcomes as the RCT with a study period up to 1 year prior to RCT initiation. The outcome 
measures and the analysis approach are described in detail in the Data Analysis section. 
 
 
V. DATA ANALYSIS 
All primary and secondary outcomes will be compared between the intervention group and 
control group. Retrospective data up to one year prior to RCT initiation may be included as a 
historical control.  
Alert-related outcomes: The primary outcome is the proportion of times a decision rule alert 
resulted in use of the decision rule by the provider in the MDCalc Connect app. This will be a 
composite calculation of all the individual clinical decision rules included in the study. The 
proportion of uses of individual clinical decision rules will serve as secondary outcomes and are 
listed below: 
- Canadian CT Head Rule 
- Canadian C-spine Rule 
- HEART Score 
- Wells’ Criteria for Pulmonary Embolism 
- PERC Rule for Pulmonary Embolism 
- Wells’ Criteria for DVT 
 

 
We plan to include additional alert-related secondary outcomes such as app usage measures from 
the MDCalc Connect app logs (e.g. total app usages across the institution, total app usages for 
providers participating in the study, etc.). The same app usage measures from one year prior to 
the study period will also be analyzed as historical controls. 
Process outcomes: The following process measures related to the decision rules will be included 
as secondary outcomes to analyze if/how the alerts and decision rule use affected provider test 
ordering habits. 
1. Advanced imaging studies: Number of advanced imaging studies ordered that are 
relevant to decision rules included in the study. Specific advanced imaging studies will 
include: CTA chest w/ IV contrast, CT chest w/ contrast pulmonary arteries for 
pulmonary embolism, Head CT without IV contrast, CT C-spine without IV contrast and 
lower extremity duplex venous scans. 
2. Laboratory tests: Number of lab tests completed relevant to the decision rules alerts. 
Specific lab tests will include: d-dimer and troponin. 
Patient outcomes: The following patient outcomes will be included as secondary outcomes to 
analyze if/how the decision rule alerts affected clinical care: 
- 30-day emergency department bounceback rates (i.e. a return ED visit within 30 days) 
- Admission rates 
 
Statistical power: 
To estimate statistical power, we developed a simulation in R. We made the following 
assumptions for these calculations: 1) the number of eligible providers and patients will be ~50 
and ~4000 (respectively) in a 4-month study period 2) the MDCalc Connect app use rate 
will increase from a baseline of 5% to 20% with the decision rule alert intervention and 3) the 
intra-class correlation coefficient (ICC) is 0.3. The ICC estimate was based off of published 
literature on the subject.15 All the estimated statistical powers were calculated based on two-sided 
t-tests with a significance level of alpha = 0.05. With a 4-month study period, we estimate more 
than 82% power to detect a difference between the intervention and control groups for the 
primary outcome (calculator usage). 
 
 
VI. DATA AND SAFETY MONITORING PLAN (if applicable) 
Not applicable. 
 
 
VII. STUDY LIMITATIONS 
The primary limitation of this study is that supervising (“attending”) physicians are not included 
in the randomization. Due to the staffing design at the study site, it was impossible to design a 
randomized controlled trial that included all providers (attendings, residents and ACPs). As such, 
attending physicians were excluded. Given that decisions are typically made jointly between the 

 
attending and junior team member (resident or APC), we suspect this may impact the results of 
this study. 
 
VIII. ETHICAL CONSIDERATIONS 
Informed consent for participating providers and patients has been waived by the University of 
Utah IRB Committee. The risks for this study are considered no more than minimal. Benefits 
include: improved quality of patient care, improved diagnostic yield of imaging studies and 
decreased the radiation exposure and cost of care for patients. There will be no financial costs to 
study participants and no compensation will be provided. 
 
 
IX. PLANS FOR DISSEMINATION OF FINDINGS 
Findings will be submitted to a peer-reviewed journal upon study completion. 
 
 
X. REFERENCES 
 
1.  
Kocher KE, Meurer WJ, Fazel R, Scott PA, Krumholz HM, Nallamothu BK. National 
trends in use of computed tomography in the emergency department. Ann Emerg Med. 
2011;58(5):452-462.e3. doi:10.1016/j.annemergmed.2011.05.020 
2.  
American College of Emergency Physicians. American College of Emergency Physicians: 
Ten Things Physicians and Patients Should Question. Choosing Wisely. Published 2014. 
https://www.choosingwisely.org/societies/american-college-of-emergency-physicians/ 
3.  
Raja AS, Greenberg JO, Qaseem A, Denberg TD, Fitterman N, Schuur JD. Evaluation of 
patients with suspected acute pulmonary embolism: Best practice advice from the Clinical 
Guidelines Committee of the American College of Physicians. Ann Intern Med. 
2015;163(9):701-711. doi:10.7326/M14-1772 
4.  
Buchanan I, Teeples T, Carlson M, Steenblik J, Bledsoe J, Madsen T. Pulmonary 
Embolism Testing Among Emergency Department Patients Who Are Pulmonary 
Embolism Rule-out Criteria Negative. Acad Emerg Med. 2017;24(11):1369-1376. 
doi:10.1111/acem.13270 
5.  
Alhassan S, Sayf A, Arsene C, Krayem H. Suboptimal implementation of diagnostic 
algorithms and overuse of computed tomography-pulmonary angiography in patients with 
suspected pulmonary embolism. Ann Thorac Med. 2016;11(4):254-260. 
doi:10.4103/1817-1737.191875 
6.  
Venkatesh AK, Kline JA, Courtney DM, et al. Evaluation of Pulmonary Embolism in the 
Emergency and Consistency With a National Quality Measure: Quantifying the 
opportunity for improvement. Arch Intern Med. 2012;172(13):1028-1032. 
doi:10.1001/archinternmed.2012.1804.Evaluation 
7.  
Ip IK, Raja AS, Gupta A, Andruchow J, Sodickson A, Khorasani R. Impact of clinical 
decision support on head computed tomography use in patients with mild traumatic brain 

 
injury in the ED. Am J Emerg Med. 2015;33(3):320-325. doi:10.1016/j.ajem.2014.11.005 
8.  
Sharp AL, Huang BZ, Tang T, et al. Implementation of the Canadian CT Head Rule and 
Its Association With Use of Computed Tomography Among Patients With Head Injury. 
Ann Emerg Med. 2018;71(1):54-63.e2. doi:10.1016/j.annemergmed.2017.06.022 
9.  
Richardson S, Cohen S, Khan S, et al. Higher Imaging Yield When Clinical Decision 
Support Is Used. J Am Coll Radiol. 2020;17(4):496-503. doi:10.1016/j.jacr.2019.11.021 
10.  
Bookman K, West D, Ginde A, et al. Embedded Clinical Decision Support in Electronic 
Health Record Decreases Use of High-cost Imaging in the Emergency Department: 
EmbED study. Acad Emerg Med. 2017;24(7):839-845. doi:10.1111/acem.13195 
11.  
Yan Z, Ip IK, Raja AS, Gupta A, Kosowsky JM, Khorasani R. Yield of CT Pulmonary 
angiography in the emergency department when providers override evidence-based 
clinical decision support. Radiology. 2017;282(3):717-725. 
doi:10.1148/radiol.2016151985 
12.  
Geeting GK, Beck M, Bruno MA, et al. Mandatory assignment of modified wells score 
before CT angiography for pulmonary embolism fails to improve utilization or percentage 
of positive cases. Am J Roentgenol. 2016;207(2):442-449. doi:10.2214/AJR.15.15394 
13.  
Wang RC, Bent S, Weber E, Neilson J, Smith-Bindman R, Fahimi J. The Impact of 
Clinical Decision Rules on Computed Tomography Use and Yield for Pulmonary 
Embolism: A Systematic Review and Meta-analysis. Ann Emerg Med. 2016;67(6):693-
701.e3. doi:10.1016/j.annemergmed.2015.11.005 
14.  
Westafer LM, Kunz A, Bugajska P, et al. Provider Perspectives on the Use of Evidence-
based Risk Stratification Tools in the Evaluation of Pulmonary Embolism: A Qualitative 
Study. Acad Emerg Med. 2020;27(6):447-456. doi:10.1111/acem.13908 
15.  
Campbell M, Grimshaw J, Steen N, Professional C. Sample size calculations for cluster 
randomised trials. J Health Serv Res Policy. 2000;5(1):12-16. 
 
XI. Appendices 
A. List if qualifying chief complaints: 
 
Canadian CT Head 
Fall 
Motor Vehicle Accident 
Motor Vehicle Crash 
Motorcycle Crash 
Motorcycle vs pedestrian 
LOC (loss of consciousness) 
Head Injury 
Headache - Injury 

 
Head injury - closed 
Trauma 
Canadian C-spine 
Fall 
Motor Vehicle Accident 
Motor Vehicle Crash 
Motorcycle Crash 
Motorcycle vs pedestrian 
Neck pain - injury 
Neck injury 
Trauma 
HEART Score 
Chest Pain 
Chest pressure 
Shortness of Breath 
Breathing Problems 
Difficulty breathing 
Wells' PE Crietria and PERC 
Rule 
Chest Pain 
Shortness of Breath 
Breathing Problems 
Difficulty breathing 
Hypoxemia 
Hemoptysis 
Wells' DVT Criteria 
Leg Pain 
Deep Vein Thrombosis 
Leg Swelling 
Swelling, Lower Extremity 
Leg Pain - Other