Communicating Narrative Concerns Entered by RNs (CONCERN) Clinical Decision Support (CDS) System
Phase: NA
Conditions: Hospital Acquired Condition
Sponsor: Columbia University

1 
 
The CONCERN Early Warning System: 
Study Protocol and Analysis Plan 
NCT03911687 
 
INVESTIGATORS 
Kenrick D. Cato, PhD, RN and Sarah C. Rossetti, PhD, RN 
PROTOCOL TITLE 
Communicating Narrative Concerns Entered by RNs (CONCERN): Clinical Decision Support 
Communication for Risky Patient States 
FUNDING NINR 
R01NR016941 
VERSION DATE 
November 11, 2016: 
 
Table of Contents 
Content 
Pages 
I. 
Original Study Protocol and Statistical 
Analysis Plan (submitted to/funded by NINR 
and IRB approved) 
2-4 
 
 
 

2 
 
1. Original Study Protocol and Analysis Plan 
Aim 3 below is focused on the CONCERN Early Warning Score Trial Design and Analysis Plan 
and is excerpted from R01 grant funded in 2017 (NINR 1R01NR016941, COmmunicating 
Narrative Concerns Entered by RNs (CONCERN): Clinical Decision Support Communication for 
Risky Patient States). Protocol below was IRB approved November 2016. 
Aim 3. Evaluate the impact of the CONCERN SMARTapp notification system on patient 
outcomes, for the primary outcomes of in-hospital mortality and length of stay (LOS) and 
secondary outcomes of cardiac arrest, unanticipated transfers to the intensive care unit, and 30-
day hospital readmission rates. 
CONCERN 
implementation 
processes will use 
open source tools, 
APIs (application 
program interfaces), 
web services, and 
SMART tools 
developed at Harvard 
Medical School. The 
need for site specific, local EHR configurations will be minimized using this approach. Our 
technology solution will allow for continuous refinement of algorithm, pushing out a new version 
every 3 months to provide consistent amount of time to evaluate and track outcomes for each 
version.  
CONCERN Intervention Trial Design will be a multiple time-series intervention (see Table 2).  
Please also refer to Table 1 for study sites and samples.  Baseline data will be collected at all 
study sites. Silent release mode (no SMARTapp notification) will be used in non-equivalent 
control units and as a post-intervention unit control to evaluate if notifying clinicians can 
decrease rates of length of stay on non-ICU units and rates of 30-day hospital readmissions. In 
Table 2 the different versions indicate dynamic, adaptive functionality. The adaptive function 
utilizes Dr. Alber’s inpatient acuity calculation118 to determine if the pattern of nursing 
documentation has changed. We have built in time for a “burn-in” phase to evaluate adoption 
and adaptation to our algorithm. 
Table 1. Sample and Setting Calendar Year 2014 (all adult patients) 
Site 
Beds Occupancy Rate In-patient Mortality 30-day Readmit 
PHS-BWH 777 95% 
1001 
18% 
PHS-NWH 300 94% 
312 
10% 
NYP-CUMC 745 95% 
1261 
18% 
NYP-Allen 300 75% 
98 
10% 

3 
 
 
CONCERN intervention trial data collection and analysis includes collection of pre and post-
intervention data during the 6 months before the intervention and 12 months after the 
intervention (see Error! Reference source not found. for timeline). Prior to conducting 
hypotheses tests, descriptive statistics will be used to describe outcome variables and key 
confounding variables. We propose a generalized linear mixed model to examine the impact of 
the CONCERN system on each of the two primary outcomes: in-hospital mortality and length of 
stay. This model is used to deal with combined data from multiple sites and can account for 
changes over time. This model allows different baselines and trajectories for different ICUs and 
non-ICU settings and can include both patient level and unit level covariates. We will also use a 
generalized linear mixed model for an interrupted time series analysis for unit level analysis. We 
will include a variable for closed versus open units in our analysis. We estimated statistical 
power for the comparison of mortality rates between the intervention and non-intervention 
periods and between silent and active model periods. All power calculations were based on 2-
sided tests with alpha = 0.05. Using hospital patients’ statistics, we expected at least 2,000 total 
admissions per month (ranging from 38 to 270 admissions in different units) with a mean 
mortality rates of 37.5 deaths per 10,000 inpatient days (ranging from 11.9 to 48.8), see Table 1. 
Using a very conservative number of total 50,000 inpatient days in each intervention period, we 
will have at least 80% statistical power to detect a difference of less than 1% relative difference 
in mortality rates. Because the 1% change in mortality are smaller than a clinically meaningful 
difference, we should have sufficient sample size   When examined by different campuses, 
using a very conservative value of inpatient days in one campus of 6000 inpatient days in each 
intervention period, we will have at least 80% statistical power to detect a difference of less than 
2% relative difference in mortality rates. There is no consensus regarding the best method for 
analyzing length of stay.  Length of stay has been analyzed using both Poisson models (or NB, 
or other related models such as Zero-truncated Poisson models) and survival models (such as 
Cox PH models).119  Both approaches will be applied to length of stay and the better performing 
model will be retained.  We will use the logit link function for 30-day hospitalization outcomes. 
We will calculate the Number Needed to Treat during the months when the CONCERN system 
is on silent mode and in active mode for the non-ICU versus ICU and at each site for the 
Table 2.  Trial Design with Multiple Time-series Intervention 
Study Arm 
Site Pre-
intervention (6 
months) 
XXX 
Phase 1 
(3 
months) 
[burn-in 
phase] 
XXX 
Phase 2a 
(3 months) 
Phase 2b 
(3 months) 
XXX 
Phase 3 
(3 
months) 
Control 
Groups 
Site 
A 
B 
V1 silent 
V2 silent 
V3 silent 
V3 silent 
Site 
B 
B 
V1 silent 
V2 silent 
V3 silent 
V3 silent 
Intervention 
Group 
Site 
A 
B 
V1 active 
V2 active 
V3 active 
V3 silent 
Site 
B 
B 
V1 active 
V2 active 
V3 active 
V3 silent 
B = baseline data; none = no intervention; silent = CONCERN App will function but will not 
display to clinician, active = CONCERN App will display to clinician. V1=version 1; V2=version 
2 refined based on continuous monitoring of data 

4 
 
prevention of each of the following outcomes: length of stay, and 30-day hospital readmission 
rates. Additional secondary outcomes include 30-day hospital readmission, cardiac arrest, and 
unanticipated transfers to the ICU as well as analytics of CONCERN system log-files for 
clinician usage metrics. The following statistical tests will assess evaluate the reliability and 
validity of the results and effects of the CONCERN intervention: mutual information, association 
statistics including t-test and regression analysis, sub-sampling, and cross-validation and control 
of confounders (e.g., patient acuity with Charlson’s Comorbidity Index, Dr. Alber’s inpatient 
acuity calculation118, and other validated acuity tools).3,45,57 
References from Original Protocol and Statistical Analysis Plan (NINR 1R01NR016941) 
3. 
Collins S, Cato K, Albers DJ, et al. Relationship Between Nursing Documentation and 
Mortality. Am J Crit Care. 2013;22(4):306-313. 
45.  
Albers DJ, Hripcsak G. A statistical dynamics approach to the study of human health 
data: resolving population scale diurnal variation in laboratory data. Phys Lett A. 
2010;374(9):1159-1164. doi:10.1016/j.physleta.2009.12.067. 
57.  
Goodwin L. Data mining issues and opportunities for building nursing knowledge. J 
Biomed Inform. 2003;36(4-5):379-388. doi:10.1016/j.jbi.2003.09.020. 
118.  Hagar Y, Albers D. Survival analysis with electronic health record data: Experiments with 
chronic kidney disease. Stat Anal Data Min ASA Data Sci J. 2014;7(5):385-403. 
119.  Austin P, Rothwell D, Tu J. A Comparison of Statistical Modeling Strategies for Analyzing 
Length of Stay after CABG Surgery. Heal Serv Outcomes Res Methodol. 2002;3(2):107-
133.