Realtime Streaming Clinical Use Engine for Medical Escalation
Phase: NA
Conditions: Clinical Deterioration, Hospital Medicine, Monitoring, Physiologic
Sponsor: Icahn School of Medicine at Mount Sinai

Study Protocol and Statistical Analysis Plan 
Protocol for: Real-time Machine Learning Alerts to Prevent Escalation of Care: A 
Pragmatic Clinical Trial 
NCT Number: NCT04026555 
Document Date: 07/28/2022 
This supplement contains the following items: 
1. Original protocol, which is also the final protocol. No changes to the protocol 
occurred during the trial. 
2. Original statistical analysis plan, final statistical analysis plan, summary of 
changes. 
 
 
 

Thursday, October 3, 2019 at 9:49:52 AM Eastern Daylight Time
Page 1 of 2
Subject:
APPROVAL OF RESEARCH GCO 19-0729
Date:
Tuesday, June 18, 2019 at 8:13:13 AM Eastern Daylight Time
From:
Richmond, Megan
To:
Levin, MaMhew
CC:
Zhao, Shan (MSH), Kia, Arash
AHachments: image001.jpg
APPROVAL OF RESEARCH
Date: 6/18/2019
To: Matthew Levin, (matthew.levin@mssm.edu)
         On 6/17/2019, an Institutional Review Board of the Mount Sinai School of Medicine, in
accordance with Mount Sinai’s Federal Wide Assurances (FWA#00005656, FWA#00005651) to the
Department of Health and Human Services approved the following human subject research from
6/17/2019 until 6/3/2020 inclusive:
Type of Review:
Initial Request for Approval
Project Title:
ReSCUE-ME
Investigator:
Matthew Levin (Dept: AN - Anesthesiology)
Project Information:
HS#: 19-00318
GCO#1: 19-0729(0001) ISMMS
Sites:
Mount Sinai
IND or IDE (if any):
No INDs;No IDEs;
Submission Details (if
any):
None
         Between 4/17/2020 and 4/22/2020, or within 30 days prior to study close, whichever is earlier,
you are to submit a completed FORM HRP-212: Continuing/Final Review Progress Report and
required attachments, in order to request continuing IRB approval or study closure. If IRB continuing

Icahn School of Medicine at Mount Sinai
Mount Sinai Beth Israel
Mount Sinai Brooklyn
The Mount Sinai Hospital
Mount Sinai Queens
New York Eye and Ear Infirmary
of Mount Sinai
Mount Sinai St. Luke's
Mount Sinai West
Program for the Protection
of Human Subjects
Institutional Review Boards
Mount Sinai Health System
One Gustave L. Levy Place, Box
 1081
New York, NY 10029-6574
T 212-824-8200
F 212-876-6789
irb@mssm.edu
icahn.mssm.edu/pphs
Initial Application
IRB-19-01700
Matthew Levin
Table of Contents
1. Summary - Title.....................................................................................................................................................................1
2. Summary - Setup..................................................................................................................................................................2
3. Summary - Background......................................................................................................................................................4
4. Research Personnel.............................................................................................................................................................6
5. Sites.......................................................................................................................................................................................... 7
6. Subjects - Enrollment.......................................................................................................................................................... 8
7. Subjects - Setting and Resources...................................................................................................................................9
8. Subjects - Populations......................................................................................................................................................10
9. Subjects - Participation.....................................................................................................................................................11
10. Subjects - Risk and Benefits........................................................................................................................................12
11. Procedures - Narrative...................................................................................................................................................14
12. Procedures - Genetic Testing......................................................................................................................................16
13. Procedures - Details....................................................................................................................................................... 17
14. Procedures - Instruments..............................................................................................................................................18
15. Procedures - Compensation.........................................................................................................................................19
16. Consent - Waiver of Informed Consent....................................................................................................................20
17. Data - Collection...............................................................................................................................................................21
18. Data - HIPAA.................................................................................................................................................................... 22
19. Data - Storage.................................................................................................................................................................. 23
20. Data - Safety Monitoring................................................................................................................................................26
21. Financial Administration................................................................................................................................................. 27
22. Attachments....................................................................................................................................................................... 28

IRB-19-01700
Initial Application
Matthew Levin
1. Summary - Title
Protocol Title
ReSCUE-ME
Principal Investigator
Matthew Levin
When the application is complete, it will be sent to the PI for submission
When the application is complete, it will be sent to the PI for submission
Primary Department
Anesthesiology
When the application is complete, it will be sent to the PI for submission
Application Initiated By
Shan Zhao
Lay Summary
The escalation of care for patients in a hospitalized setting between nurse practitioner managed services, teaching
 services, step-down units, and intensive care units is critical for appropriate care for any patient. Often such
 "triggers" for escalation are initiated based on the nursing evaluation of the patient, followed by physician history
 and physical exam, then augmented based on laboratory values. These "triggers" can enhance the care of patients
 without increasing the workload of responder teams. One of the goals in hospital medicine is the earlier identification
 of patients that require an escalation of care. We have developed a model through a retrospective analysis of the
 historical data from the Mount Sinai Data Warehouse (MSDW), which can provide machine learning based triggers
 for escalation of care (Approved by: IRB-18-00581). This model is called "Medical Early Warning Score ++" (MEWS
++). This IRB seeks to prospectively validate the developed model through a pragmatic clinical trial of using these
 alerts to trigger an evaluation for appropriateness of escalation of care on two general inpatients wards, one medical
 and one surgical.  These alerts will not change the standard of care. They will simply suggest to the care team that
 the patient should be further evaluated without specifying a subsequent specific course of action. In other words,
 these alerts in themselves does not designate any change to the care provider's clinical standard of care. We
 estimate (see "Narrative" section for details) that this study would require the evaluation of ~ 18380 bed movements
 and approximately 30 months to complete, based on the rate of escalation of care and rate of bed movements in the
 selected units (9W, 9E, 10W, 10E).
IF Number
IF2411760
Page 1 of 28

IRB-19-01700
Initial Application
Matthew Levin
2. Summary - Setup
Funding Has Been Requested /
 Obtained
No
Application Type
Request to Rely on Mount Sinai IRB
Research Involves
Prospective Study ONLY
Consenting Participants
No
Humanitarian Use Device (HUD)
 Used Exclusively in the Course of
 Medical Practice
No
Use of an Investigational Device to
 Evaluate Its Safety or Effectiveness
No
Banking Specimens for Future
 Research
No
Cancer Related Research that
 Requires Approval from the
 Protocol Review and Monitoring
 Committee (PRMC).
No
   Is this Cancer Related Research?  Cancer Related Research is defined as research that
 has cancer endpoints or has a cancer population as part of or all of its targeted population.
 This includes protocols studying patients with cancer or those at risk for cancer.
  
Clinical Trial
Yes
   
   * A prospective biomedical or behavioral research study of human subjects that is
 designed to answer specific questions about biomedical or behavioral interventions
 (drugs, treatments, devices, or new ways of using known drugs, treatments, or devices). 
   *  Used to determine whether new biomedical or behavioral interventions are safe,
 efficacious, and effective.    
  
Drugs / Biologics 
No
   
   * Drugs / Biologics That Are Not a Part of Standard Practice 
   * Controlled Substances 
   * Drugs / Biologics Supplied by the Research Sponsor or Purchased with Study Funds 
  
   Ionizing Radiation for imaging or therapy, including X-Ray, Fluoroscopy, CT, Nuclear
 Medicine, PET andor Radiation Therapy:
  
* Purely for standard of care:
No
* In frequency or intensity that
 exceeds what is necessary for
 standard of care:
No
Hazardous Materials
No
   * Recombinant DNA 
   * Viral Vectors 
   * Plasmids 
   * Bacterial Artificial Chromosomes 

   * Toxic Chemicals, Potentially Toxic Medications, Carcinogens 
   * Autologous Cell Lines 
  
Request Use of Clinical Research
 Unit Resources
No
Page 3 of 28

IRB-19-01700
Initial Application
Matthew Levin
3. Summary - Background
Objectives
Mount Sinai Hospital has developed a Rapid Response Team (RRT) system designed to give general floor care
 providers additional support for patients who may be requiring a higher level of care. This system enables both
 nurses and physicians to notify the RRT and have a critical care team evaluate the patients. During the period of
 03/01/2018 to 09/17/2018, Mount Sinai Hospital floor units on 10W and 10E units made 357 rapid response team
 (RRT) calls with only 58 leading to an actual increase in the level of care (true positive rate ~ 16%). Similarly, the
 Electronic Health Record (EHR) generated 839 sepsis Best Practice Alerts (BPAs) yet only five led to escalations in
 care (true positive rate ~ 0.5%). The results above would imply that over 168 evaluations need to be made to identify
 a single case where the patient required an escalation in care. The goal of ReSCUE-ME is to evaluate prospective
 model performance and identify the best spot which we can incorporate MEWS++ into RRT and Primary providers
 workflow. The primary endpoint is rate of escalation of care on 10W and 10E during the study period. 
Background
In a prior study, our group has demonstrated that a machine learning model (MEWS++) significantly outperformed
 a standard, manually calculated MEWS score on a large retrospective cohort of hospitalized patients. To develop
 this model, we used a data set (Approved by: IRB-18-00581) of 96,645 patients with 157,984 hospital encounters
 and 244,343 bed movements. We found that MEWS++ was superior to the standard MEWS model with a sensitivity
 of 81.6% vs. 44.6%, specificity of 75.5% vs. 64.5%, and area under the receiver operating curve of 0.85 vs. 0.71.
 Encouraged by this prior result, we are seeking to evaluate our model in a prospective study.
A silent pilot of the ReSCUE-ME alerts has been running on 10E and 10W since Feb 2019. We have been
 continuously monitoring the alert performance via a real-time web-based dashboard. The results are summarized
 below: 
• Median # of alerts to primary team, per floor, per day: 8 
• Median # of alerts to RRT, per floor, per day: 4 
• Sensitivity 0.76, Specificity 0.68, AUC 0.77
• Accuracy 0.69, Precision 0.3, F1 Score 0.43 
This performance compares very favorably to the performance seen in the retrospective historical cohort used to
 develop the MEWS++ model: 
• Sensitivity 0.82, Specificity 0.76, AUC 0.85 
• Accuracy 0.76, Precision 0.12, F1 Score 0.19"
Primary and Secondary Study Endpoints
Primary Endpoints:
• Composite of the Rate of escalation of care (from floor to Stepdown, Telemetry, ICU) and rate of RRT initiated
 therapy (including but not limited to blood pressure support, respiratory care support, anti-biotic augmentation,
 invasive monitoring) in the selected units (10E and 10W)
Secondary Endpoints: 
• Blood pressure support utilization (e.g., initiation of vasopressor medication, administration of fluid bolus)
• Respiratory care support utilization (e.g., nasal cannula to high flow)
• Accuracy of predicting care escalation from the floor to ICU, step-down, and Telemetry
• Cardiac arrest/mortality rate in 10E and 10W compared with two control units (9W and 9E)
• Evaluate the average calls per day and average calls per patient when compared to historical and matched general
 medical-surgical floor units (9W and 9E)
• Evaluate through a survey format of nursing and physician care providers the utility of MEWS++ scoring in
 determining whether or not an RRT call should be placed. This survey will be administered at the beginning and end
 of the enrollment period through an online REDCap form hosted within our institution (or comparable system). The
 survey will be administered by one of the co-investigators within the study.
Protocol Was Already Approved
 by the Icahn School of Medicine at
 Mount Sinai (ISMMS) Institutional
 Review Board (IRB) Under a
 Different Principal Investigator
No

Protocol Was Previously Submitted
 to an External(non-ISMMS) IRB
No
Page 5 of 28

IRB-19-01700
Initial Application
Matthew Levin
4. Research Personnel
Name/Department
Role/Status
Contact
Access
Signature
 Authority
Phone
Email
Matthew Levin /
 Anesthesiology
PI / Faculty
Yes
SIGNAUTH
Shan Zhao /
 Pharmacology
 and Systems
 Therapeutics
Co-Investigator / 
Yes
EDIT
310-751-4604
Rohit Gupta / Surgery Co-Investigator / 
EDIT
Jennifer Wang /
 Surgery
Co-Investigator / 
READONLY
Robert Freeman /
 Office of Clinical
 Research
Co-Investigator / 
READONLY
Roopa Kohli-Seth /
 Surgery
Co-Investigator / 
EDIT
(212)
 241-8867
Sanam Ahmed /
 Surgery
Co-Investigator / 
READONLY
212-241-8867
Mirhadi Arash Kia / IT
 Department
Co-Investigator / 
Yes
EDIT
Prem Timsina / IT
 Department
Project Director / 
READONLY
Surafel Tsega /
 Anesthesiology
Co-Investigator / 
READONLY
Benjamin Kummer /
 Neurology
Co-Investigator / 
READONLY
Michael Kitz /
 Anesthesiology
Co-Investigator / 
READONLY
 
Page 6 of 28

IRB-19-01700
Initial Application
Matthew Levin
5. Sites
Site Name
The Mount Sinai Hospital
Other External Site Name
Contact Details
Approved
Approval Document
Funded By Mount Sinai
Other IRB
Page 7 of 28

IRB-19-01700
Initial Application
Matthew Levin
6. Subjects - Enrollment
Site Name
The Mount Sinai Hospital
Subjects To Be Enrolled
18680 
Total Number of Subjects to be
 Enrolled Across All Listed Sites
 Above (Auto Populated)
18680
Page 8 of 28

IRB-19-01700
Initial Application
Matthew Levin
7. Subjects - Setting and Resources
Setting of Human Research
Other
Specify Other Setting of Human Research
Hospital Units on 10W, 10E, 9W, 9E.
Total Number of Subjects Needed
 To Complete Study
18680
Feasibility of Meeting Recruitment Goals
Our initial observation shows the rate of escalation ~ 10% for the selected units (10W and 10E). The estimated
 sample size is ~ 934 escalations of care which we expect to observe by including ~9340 bed movements. The
 estimated bed movements per day in the selected units is ~ 10 which makes the expected pilot days ~ 934 days
 (~ 2.5 yrs.). We anticipate an equal number of escalations for 9W and 9E, which will not receiving any MEWS++
 alerts.#
Facilities To Be Used for Conducting Research
Hospital Units on 9W, 9E, 10W, 10E
Multi-Center Study
No
Community-Based Participant
 Research Study
No
  PI must attest to the following. 
   * Process is adequately described to ensure that all persons assisting with the trial are
 adequately informed about the protocol, the investigational product(s), and their trial-
related duties and functions.
     
    
Page 9 of 28

IRB-19-01700
Initial Application
Matthew Levin
8. Subjects - Populations
Inclusion Criteria
All patients age 18 or greater who were admitted to a general care unit on 9W, 9E, 10W, 10E. 
Exclusion Criteria
Any admitted patient who has a "Do Not Resuscitate (DNR)" and/or a "Do Not Intubate (DNI)" order in the EHR, any
 patient made "level of care" by RRT as documented in REDCap.
Enrollment Restrictions Based
 Upon Gender, Pregnancy,
 Childbearing Potential, or Race
No
Age Range(s)
18 to 64 Years, 65 Years and Over
Targeted Population(s)
Adults - Patients 
Other Aspects that Could Increase Subjects Vulnerability
Prisoners and non-laboring pregnant women will be included if they are admitted to 10E/W or 9E/W. This will be rare
 but possible. Incapacitated patients will be provided the study information sheet within their paper chart and included
 unless their health care proxy/legally authorized representative chooses to opt them out of the study. If they were to
 regain consciousness and capacity, then they would have the ability to withdraw from the study."
Safeguards to protect Subjects rights and welfare
All key personnel have been HIPAA trained. All data will be encrypted and stored on secure servers within the
 hospital data center, with access restricted based on involvement in the project. Patients will also be offered multiple
 methods of withdrawing participation consent from the study, which includes by phone, email, website, and nursing
 care.
Page 10 of 28

IRB-19-01700
Initial Application
Matthew Levin
9. Subjects - Participation
Duration of an Individual Subjects Participation in the Study
During the entirety of their care on 9W, 9E, 10W, 10E
Duration Anticipated to Enroll All Study Subjects
We anticipate, given the current volume of bed movements, that it will take 30 months to complete enrollment.
Estimated Date for the Investigators
 to Complete This Study
Within 3-5 years
Procedures for Subjects to Request Withdrawal
Subjects may also withdraw from permission for the use and disclosure of any of your protected information for
 research and can do so by contacting the unit business associate "BA", by email to arash.kia@mssm.edu or by
 calling to (646) 605-4916 with the information specified in "Waiver  Information Sheet" (Appendix 1).
Procedures for Investigator to Withdraw Subjects
The Principal Investigator may stop subject involvement in this research study at any time without subject consent.
 This may be because the research study is being stopped, the instructions of the study team have not been followed,
 the investigator believes it is in their best interest, or for any other reason.
"Not participate" or "withdraw" means the alerting will be suppressed/turned off for the patient, their data will not be
 collected, and no alerts will be sent. Any data or alerts prior to withdrawal will be retained. This information has been
 added to the consent.
Participants Will Be Recruited
Yes
Recruitment Method(s)
Clinical Practice
How Participants Will Be Identified
All patients admitted to 10E, 10W, 9E, and 9W
Who Will Initially Approach
 Potential Participants
Clinic Personnel
How Research Will Be Introduced to Participants
Description of the study protocol will be given to participants upon admission to the floor. We will instruct the
 Business Associate and nursing team members to specifically call to attention to the provided information sheet
 when a patient is admitted to the unit. We will also post prominent printed notices describing the study throughout
 the unit (e.g. on the walls near the nursing desk, on any bulletin boards designed for patient communication, etc).
 Furthermore, if a patient does trigger an alert, the RRT staff will be instructed to notify the patient of the information
 sheet. Finally, upon discharge, the information sheet would be provided again to the patient as part of their check-
out. As such, there exists at least 3 touch points for the patient to discuss the study with a care provider.
How Participants Will Be Screened
Automated screening by the software that generates alerts.
Page 11 of 28

IRB-19-01700
Initial Application
Matthew Levin
10. Subjects - Risk and Benefits
Risks to Subjects
The MEWS++ alerts will serve to notify the care team of a potential deterioration in a patient's clinical condition. What
 the provider may choose to do with the MEWS++ alert is left completely to their discretion and we do not propose
 any determination of care by the MEWS++ alert. As a result, there will be no alteration in the standard of care. All
 therapeutic interventions will be at the discretion of the care providers and the expected risk of monitoring for select
 individuals within the interventional cohort should be minimal. The potential risk of breach of confidentiality or loss
 of protected healthcare information (PHI), will be unlikely as all data will be password protected and any patient
 identifiers will be encrypted; furthermore, all data gathering will be done on Mount Sinai internal network computers
 and devices.#
We do not change any workflow for the primary team and RRT. We just increase awareness in the clinicians' mind
 that a patient may be in danger of deterioration. The primary intervention by the Primary team is simply to increase
 the frequency of vitalsigns measurement. This will not be associated with any risk or patient safety concerns. We will
 not remove any of current notifications being received by RRT we will just add one more notification to the current
 workflow. 
One possible risk is alert fatigue. The primary cause of alert fatigue for the RRT team is currently the "Sepsis BPA"
 generated by Epic, not clinician generated alerts or the alerts that will be generated by this study. The alerting
 protocol attached to the application was carefully designed to minimize the number of additional alerts. Based on our
 modeling we anticipate 5 additional alerts to RRT per additional 12-hour shift. This is confirmed by the silent pilot.
 This was extensively discussed with both nursing and RRT leadership who agreed that the burden of this additional
 alerting is tolerable. The Sepsis BPA's generate far more alert fatigue. They are being modified to fire less frequently,
 as part of a non-IRB approved PI/QI changes being made directly by the CMO's office and Epic implementation
 team, independent of this protocol. 
In order to minimize alert fatigue from our alerts, we will be continuously trending the alert burden of the study via the
 web dashboard and other custom analytic tools. In addition, care provider participants will be polled for their degree
 of alert fatigue. A review committee will be formed to assess the need to adjust the degree of alerting. 
Currently the RRT team and floor nurses are overburdened by Sepsis BPA alerts. Our alerts are more sensitive and
 more specific and are carefully calibrated to fire less often based on both the retrospective data used to develop
 the alert and our silent pilot. c. Since the number of alerts is fairly low (5 alerts per 12 hours per floor) and the
 refractory period can be adjusted automatically based on the clinicians' judgment (see below), we don't believe we
 will encourage "over testing". 
It should be noted that over testing is often a result of inability to ignore results of alerts without perceived
 medicolegal ramification. By the very fact that our alert is investigational, it would not produce similar concerns. 
Inducing a false sense of security is perhaps unavoidable with the introduction of any new monitoring modality.
 During our education sessions with providers, it will be stressed that MEWS++ is an investigational alert and it
 should not override clinical judgement. We will have two further mechanisms for soliciting clinician feedback and
 ensuring a false sense of security is not imbued in the staff. First, we will be in close communication with nursing
 and RRT leadership throughout the pilot, via check-in meetings initially weekly, and then monthly or as needed
 throughout the study. Second, the RRT team logs details of every RRT visit to a REDCap database. These data
 are automatically integrated into our analysis pipeline. We match alerts to RRT calls as part of our performance
 monitoring. If a deterioration occurs that is missed by our algorithm but seen by RRT, will be able to analyze and
 understand what factors may have contributed to the miss, and if alert fatigue or a false sense of security contributed
 to the miss.
Description of Procedures Taken to Lessen the Probability or Magnitude of Risks 
No additional data will be gathered beyond that of the standard of care for patients, thus limiting risks of the additional
 information. The research personnel table lists all personnel who may obtain viewing privileges to the collected data.
Provisions for Research Related Harm / Injury
There is no risk of research related to harm or injury as no therapeutic intervention is made based on the results from
 this study's evaluation. All participant will receive standard of care interventions.
Expected Direct Benefit to Subjects

As a direct benefit from enrolling in this study, we expect the subjects will be more closely monitored for deterioration
 in clinical condition and will receive more immediate escalation in care when compared to both historical and
 matched until controls. 
Benefit to Society
We hope that the proposed technology will prove to have a highly sensitive and specificity for predicting individuals
 who are at risk of clinical deterioration and may require a higher/more intensive level of care. If this does prove to
 be true, this would be a significant advance to the field of medicine. It would help to reduce death within an inpatient
 hospital setting.
Provisions to Protect the Privacy Interests of Subjects
By only allowing only members of the investigational team to access PHI we protect subjects' privacy. All study
 personnel has completed HIPAA training, which ensures the utmost care is used to handle PHI.
Economic Impact on Subjects
There is no economic burden on the patient.
Page 13 of 28

IRB-19-01700
Initial Application
Matthew Levin
11. Procedures - Narrative
Description of the Study Design
Patients admitted to 10W and 10E meeting the inclusion and exclusion criteria will undergo a modified RRT
 evaluation protocol as detailed in Appendix 4: Process Flow. In brief, the patient will be evaluated based on our
 retrospectively validated machine learning algorithm (MEWS++) designed to improve identification of patients who
 may need care escalation. Patients on 9W and 9E will be followed but no alerts, interventions or changes from
 standard of care will be pursued. The study is designed to evaluate the equivalence in the proportion of escalations
 between the standard of care RRT protocol and our proposed ReSCUE-ME protocol. Patients admitted to 9W and
 9E will not undergo any additional monitoring and thus will serve as controls.
Further details of the alerting system are listed below.
a. Notifications can be suppressed or re-enabled on a per-patient basis. This is a core feature as it is needed to fulfill
 the opt-out process that we documented in our application's appendices.
b. Currently there are several automated alerts sent to Vocera. These include ventilator and telemetry alarms,
 e.g. an alarm if the ECG monitor detects asystole. There is also a simple rules-based Epic BPA for hyperkalemia
 that sends an alert to the Vocera if a patient's potassium level is high. There is an administrative policy on alarm
 management that covers these existing alerts - Clinical Alarm Management GPP-242 – link to policy below: http://
policies.mountsinai.org/web/general-policies-and-procedures/policies/-/policy-management/viewPolicy/565852/.
c.The maximum number of alerts within the default 8-hour refractory window has been set to 3 in order to minimize
 alert fatigue. The entire process was carefully modeled in cooperation with a senior operations manager from the
 hospital process improvement team, with input from the nurse managers on 10E/W and the RRT leadership. The
 refractory period can also be customized to be shorter based on RRT preference and clinical judgement. The RRT
 attending simply sends a text message back to the system via Cureator indicating the desired refractory period.
 Thus, for a very sick patient the refractory period could be set as short as 2 hours. 
d. All Nurses on 10E/W will receive in-service training prior to go-live. This is being coordinated with the nurse
 managers of 10E/W. Members of the research team will be in attendance.
e. Nurses and any other clinician/staff will always be able to call a Stroke Code/Team 7000 at any time.
f. "MEWS++ will run in addition to the current Sepsis BPA. MEWS++ is tuned for deterioration in general, not
 specifically sepsis. Both MEWS++ and the Sepsis BPA may fire for the same patient at the same time, or within a
 short interval. If a patient gets a MEWS++ alert and turns out to be Septic, they can be put on the Sepsis Pathway in
 Epic." 
Description of Procedures Being Performed
For each patient, real-time data from the EHR (Epic), the admitting system (Cerner ADT), the ECG (MUSE), and
 laboratory information systems (SCC) will be used by ReSCUE-ME to produce a MEWS++ score (between 0 to 1)
 predicting the likelihood that the patient will require escalation of care within the next 6 hours. Upon the patient being
 admitted to the unit, the patient will be evaluated based on any update in the EMR. If the prediction score exceeds
 0.64, the RRT team will be notified directly. If the score is between 0.55 and 0.64, the nursing team will be notified
 and increased nursing monitoring will be initiated. Otherwise, the patient will undergo re-evaluation with a 8-hour
 refractory window. If the patient has met criteria for increased nursing monitoring, a refractory 8-hour refractory
 window will be applied and if the MEWS++ scoring ever exceeds 0.65, the RRT team will be notified. The notification
 will be processed through Vocera, Connexal and Cureatr, which are hospital approved secure messaging systems. A
 detailed process flow diagram is provided in Appendix 4.
Description of the Source Records that Will Be Used to Collect Data About Subjects 
For each patient, real-time data from the EHR, the admitting system (Cerner ADT), the ECG (MUSE), and laboratory
 information systems (SCC) will be ingested by the ReSCUE-ME engine and used to generate a prediction score.
 Additional information provided by the RRT team on the result of the RRT intervention will be extracted from
 REDCap. These values as specified in Appendix 3 will be processed through the ReSCUE-ME data streaming
 engine to generate a MEWS++ score for patients who have not been excluded or withdrawn from the study. If an
 individual withdraws participation from the study, all data pertaining to this patient's current visit will be removed from

 ReSCUE-ME. Clinical documents may also be assessed for the purpose of further clarification of events during the
 RRT intervention.
Description of Data that Will Be Collected Including Long-Term Follow-Up
The data that will be collected during this study include vitals, electrocardiogram, laboratory data, clinical evaluations,
 and patient demographics from ADT, Epic, MUSE, and Laboratory Information System. A detailed list is provided
 in Appendix 3. At this time we do not plan on conducting long-term follow-up upon the discharge of the patient from
 their current hospital visit. 
Research Requires HIV Testing
No
Page 15 of 28

IRB-19-01700
Initial Application
Matthew Levin
12. Procedures - Genetic Testing
Genetic Testing Will Be Performed
No
Guidance and Policies > Future Use Data Sharing and Genetic Research
  
Page 16 of 28

IRB-19-01700
Initial Application
Matthew Levin
13. Procedures - Details
Surveys or Interviews
Yes
Type of Instruments Being Used
Created By Research Team
Description of Instruments Created By Research Team
A REDCap survey will be created to understand the impact of the alerts on the RRT team. This clinician survey has
 not yet been developed. We will not do this piece of the research until the survey has been approved by the IRB and
 the protocol modified appropriately and also approved.
Audio / Photo / Video Recording
No
Deception
No
Results of the Study Will Be Shared
 with Subjects or Others
No
Page 17 of 28

IRB-19-01700
Initial Application
Matthew Levin
14. Procedures - Instruments
Instruments Created By Research Team
Type
REDCap RRT survey
Name
REDCap RRT survey
Upload
REDCap RRT Survey Placeholder.docx
Page 18 of 28

IRB-19-01700
Initial Application
Matthew Levin
15. Procedures - Compensation
Compensation for Participation
No
Page 19 of 28

IRB-19-01700
Initial Application
Matthew Levin
16. Consent - Waiver of Informed Consent
Requesting Waiver For
At the request of the IRB, the principal investigator developed the following opt out process in cases where patients
 did not want to participate:
Opt-Out Process for ReSCUE-ME
1. Every patient will receive at least one printed copy of the "Waiver Information Sheet" letter at various entry points
 into the hospital unit:
a. The 9W, 9E, 10W, and 10E business associate will insert a print out of this waiver into their "welcome packet" (a
 packet of information given to all patients entering any hospital unit) as part of the current standard operating
 procedure.
2. We will offer to distribute this letter to every hospitalist through Mount Sinai Medicine Department's Division of
 Hospital Medicine.
4. The opt-out letter will be in English and Spanish.
Educating Nursing and Physicians about the Opt-Out Process
1. An educational conference will be partially devoted to the subject of introducing the study and explaining the
 scientific background and methods of the research.
a. Medicine Department's Division of Hospital Medicine
b. The Mount Sinai's Institute for Critical Care Medicine
c. Nursing Teams on the participating units.
2. All members of the hospitalist and rapid response care team (attending, residents, and nursing) will be given a
 script that will be posted on the departmental intranet to give patients information regarding the study.
3. All members of the hospitalist care team and business associates on the participating units will be given instruction
 as to the procedure for withdrawing patients from the study.
Instructions for Hospitalist Care Team:
1. If the patient has no questions regarding the ReSCUE-ME Study, then proceed as usual.
2. If the patient has questions regarding the ReSCUE-ME Study that you cannot answer, please page one of the
 investigators listed on the Clinical Data Science group's intranet link for the study. If you cannot obtain an answer
 quickly for the patient, offer the patient the option of withdrawing from the research.
3. If the patient spontaneously, or after having had questions answered, wishes to withdraw from the research, the
 provider should submit a request to the http://cds.mountsinai.org/ReSCUEMe website.
Type of Waiver
Waiver of Consent
How the Research Involves No More Than Minimal Risks to Participants
All participant subjects will receive standard of care at all times. The MEWS++ alerts will simply inform the clinicians
 that a patient may be in danger of clinical deterioration, without providing any suggestions on the best course of care.
 The most appropriate intervention will be determined by the treating clinician.
How the Waiver / Alteration Will Not Adversely Affect the Rights and Welfare of Participants
There will be no alteration to the standard of therapeutic care in the proposed study in either the interventional or non-
interventional group. As a result, we don't anticipate any detrimental changes to the welfare of the patient based on
 the MEWS++ and MEWS scoring and the potential for increased monitoring. Conversely, we expect that ReSCUE-
ME will improve patient welfare by improving care.
Why It Is Not Practical To Conduct this Research Without a Waiver or Alteration of Informed Consent
The number of potential patients that need to have consented means that obtaining consent is neither feasible nor
 practical as it would lead to an overwhelming burden on staffing resources.
Plans for Providing Participants with Additional Pertinant Information After Participation Where Appropriate
The care providers will be contacted during this enrollment period for any clinically relevant information that may
 arise during analysis, though acting on such information is at the discretion of the care provider. This contact may be
 through grand rounds presentations, in-service training, email distribution, or other means.
Page 20 of 28

IRB-19-01700
Initial Application
Matthew Levin
17. Data - Collection
Health Related Information Will Be
 Viewed, Recorded, or Generated
Yes
Description of Health Information That Will Be Viewed, Recorded, or Generated
The artificial intelligence algorithm will process and analyze data obtained from the EHR and other systems as
 previously described. The Data Collection Sheet contains the exact list of variables. The algorithm generates a
 MEWS++ score that is viewable through the EHR by the care provider team.
Non-Health Related Information Will
 Be Viewed or Recorded
No
HIV / AIDS Related Information Will
 Be Viewed or Recorded
No
Data That Will Be Viewed,
 Recorded, or Generated Contains
 ANY of the Following Directly
 Identifiable Information
Yes
Will Be Viewed
Medical Record Number
Will Be Recorded
Medical Record Number
Data Collection Sheet
DataElements.xlsx
        
        A Data Collection Sheet is required if you are either performing a retrospective review,
 or your study meets the category of exempt 4 research, or your study meets the category
 of expedited 5 research. Please upload it here.
        
    
Data Collection Source(s)
Medical Chart (Paper or Electronic), Data
 Warehouse
Page 21 of 28

IRB-19-01700
Initial Application
Matthew Levin
18. Data - HIPAA
Obtaining HIPAA Authorization
No
Requesting Waiver or Alteration
Waiver
Research Could Be Practicably
 Conducted Without Access to
 and Use of Protected Health
 Information (PHI)
No
Explanation Why Research Could Not Be Practicably Conducted Without Access to and Use of PHI
In order for a MEWS++ alert to be of value, the provider needs to know the identity of the patient for whom the alert
 was generated. Thus the alerting system needs to know the identity of the patient as well. 
Explanation Why Research Could Not Be Practicably Conducted Without a Waiver or Alteration of
 Authorization
It is neither practical nor feasible to consent over 18,000 individual patients. Additional details on why we are
 requesting a waiver can be found under the "Consent" section of this IRB application.
How PHI Will Be Protected from Improper Use or Disclosure
All PHI will be stored in secure, password protected servers that can only be accessed by authorized members of the
 study team. All study personnel have completed HIPAA and human subjects research training, which ensures utmost
 care is used to handle PHI.
PHI Will Be Destroyed at the
 Earliest Opportunity Consistent
 with the Research
Yes
When and How PHI Will Be Destroyed 
Data are stored in accordance with the data retention policy of the manuscript in which the results will be published.
 This may vary by publication but is generally 7 years. Data will be destroyed by overwriting the disk sectors used to
 store the data with 0s.
PHI Will Be Shared
No
  PI must attest to the following.
  * I assure that the protected health information (PHI) will not be disclosed to any other
 person or entity not listed on this form except where required by law or for the authorized
 oversight of this research project. If at any time I want to reuse this PHI for other purposes
 or disclose it to other individuals or entities I will seek approval from the IRB. 
  
 
Page 22 of 28

IRB-19-01700
Initial Application
Matthew Levin
19. Data - Storage
Location Where Data Will Be Stored
Secure servers within the hospital data center.
How will the data be stored?
Other
Specify How Data Will Be Stored
All data are will be stored within an encrypted, password protected database. The servers can only be accessed via
 Secure Shell or HTTP-Secure technology requiring a password or military-strength encryption key. Local system
 firewalls and operating system security patches are kept up-to-date on an ongoing basis. The access to this data will
 be reviewed on a quarterly basis and any identified breaches will be reported in accordance with HIPAA compliance.
 Data will also be encrypted during transit among the components of the real time alerting system. 
Research Personnel Responsible
 for:
Matthew Levin
Accessing Data
Yes
Receipt or Transmission of Data
Yes
Holding Code That Can Be Linked
 to Identity of Participants
No
Research Personnel Responsible
 for:
Shan Zhao
Accessing Data
Yes
Receipt or Transmission of Data
Yes
Holding Code That Can Be Linked
 to Identity of Participants
No
Research Personnel Responsible
 for:
Rohit Gupta
Accessing Data
No
Receipt or Transmission of Data
No
Holding Code That Can Be Linked
 to Identity of Participants
No
Research Personnel Responsible
 for:
Jennifer Wang
Accessing Data
No
Receipt or Transmission of Data
No
Holding Code That Can Be Linked
 to Identity of Participants
No
Research Personnel Responsible
 for:
Robert Freeman
Accessing Data
No
Receipt or Transmission of Data
No
Holding Code That Can Be Linked
 to Identity of Participants
No
Research Personnel Responsible
 for:
Roopa Kohli-Seth
Accessing Data
No
Receipt or Transmission of Data
No
Holding Code That Can Be Linked
 to Identity of Participants
No
Research Personnel Responsible
 for:
Sanam Ahmed

Accessing Data
No
Receipt or Transmission of Data
No
Holding Code That Can Be Linked
 to Identity of Participants
No
Research Personnel Responsible
 for:
Mirhadi Arash Kia
Accessing Data
Yes
Receipt or Transmission of Data
Yes
Holding Code That Can Be Linked
 to Identity of Participants
No
Research Personnel Responsible
 for:
Prem Timsina
Accessing Data
Yes
Receipt or Transmission of Data
No
Holding Code That Can Be Linked
 to Identity of Participants
No
Research Personnel Responsible
 for:
Surafel Tsega
Accessing Data
No
Receipt or Transmission of Data
No
Holding Code That Can Be Linked
 to Identity of Participants
No
Research Personnel Responsible
 for:
Benjamin Kummer
Accessing Data
No
Receipt or Transmission of Data
No
Holding Code That Can Be Linked
 to Identity of Participants
No
Research Personnel Responsible
 for:
Michael Kitz
Accessing Data
No
Receipt or Transmission of Data
No
Holding Code That Can Be Linked
 to Identity of Participants
No
Duration Data Will Be Stored
Data are stored in accordance with the data retention policy of the manuscript in which the results will be published.
 This may vary by publication but is generally 7 years.
Steps That Will Be Taken to Secure the Data During Storage, Use, and Transmission
Only the designated investigator and co-investigators as listed on this form will have access to patient data used in
 this study and will be the sole arbiters for receipt and transmission of this data. The data will be password protected
 on Mount Sinai computers. Data are encrypted both at rest (i.e., when stored in the database) and while in motion
 (i.e., during transit between various components of the alerting system) using industry standard encryption protocols.
 Data will only be accessed from within the Mount Sinai network to ensure confidentiality, which is also password
 protected and physical access restricted. Any presentation will separate the presented material from the associated
 identifiers. All individuals have completed institutionally approved HIPAA training.
Power Analysis/Data Analysis Plan (Including Any Statistical Procedures)
Sample size calculation
Based on the pilot feasibility study, the current true rate of escalation of care in the selected units is ~ 10%. We
 expect that MEWS++ will identify at least 50% of the escalations, on average ~ 6 hrs before the expected time
 of deterioration/escalation. It is estimated that RRT and front-line providers collaboratively will be able to prevent
 ~ 20% of the predicted escalations.  We are specifying a non-inferiority or superiority margin of 5%. As a result,
 the estimated sample size needing to achieve a type 1 error of 5% and power of 95% is 934. based on the rate
 of escalation ~ 10%, a total of approximately 9340-bed movements will need to be recruited for this study in the
 MEWS++ group and a total of 18680-bed movements across both groups. We assumed that there is not a significant

 difference in the escalation rate between the selected units (10E and 10W) and control units (9E and 9W). Interim
 evaluations every 6 month will be conducted and if at any point the model meets statistical significance or major
 flaws identified, a subsequent report will be submitted.
Statistical Analysis:
#The basics of generating a MEWS++ score, the sources of the data and where the results will be made available
 to the floor patient care providers is described under "Procedures > Narrative" section of this IRB. Briefly, a random
 forest model has been trained to generate a prediction score based on 36 clinical variables. This score represents
 the likelihood of clinical deterioration. For all interim reports, final study outcome report, and publication analysis
 our research team plans to use an array of parametric and non-parametric statistical approaches to evaluate the
 significances of the primary and secondary endpoints. These would include, but not limited to binomial proportion
 tests, students t-tests, Mann-Whitney U test, Fisher's exact test, and ANOVA. Additionally, we plan to examine any
 potential significant difference in the demographics of the patients admitted to 10W and 10E when compared to 9W
 and 9E, significant differences will be incorporated into a multi-variant model such as a linear/logistic regression
 model to further elucidate factors that may have confounded the # of RRT evaluation and escalation events.
Page 25 of 28

IRB-19-01700
Initial Application
Matthew Levin
20. Data - Safety Monitoring
More Than the Minimum Data
 Safety Monitoring Will Be Done
No
  
  The following minimum requirements apply to all projects, including retrospective
 reviews of medical records, use of tissue samples, and many minimal risk studies, such
 as observational and survey research. Because these minimum requirements apply to
 all studies, a specific written DSMP will not usually be required for projects that do not
 pose greater than minimal risk to subjects. The MSSM PPHS may alter the required level of
 monitoring if appropriate. 
  For all projects, the principal investigator must have a plan to assure that data integrity
 will be maintained during its collection, storage and analysis. All research projects
 must adhere to MSSM recommendations on the storage of research data. Loss of data
 containing identifiable information is reportable to the IRB within 5 business days.
  Any problems concerning the consent process and any subject complaints should be
 monitored by the investigator. Reports of such problems must be made at least annually.
 The discretion of the protocol director will guide the need to report these problems
 immediately or more frequently.
  The principal investigator is, typically, the monitoring entity for the minimum DSMP. When
 a principal investigator is not a faculty member, the supervising faculty member must be
 responsible for the data and safety monitoring aspect of the protocol.
     
 
Will the Research Include Data
 Coordinating Center Activities?
No
Page 26 of 28

IRB-19-01700
Initial Application
Matthew Levin
21. Financial Administration
  This information will help the Financial Administration of Clinical Trials Services (FACTS)
 office determine whether a Medicare Coverage Analysis (MCA) is needed for the research
 study. If you have any questions while completing this form, please contact the FACTS
 office at (212) 731-7067 or FACTS@mssm.edu.
  
 
Clinical Research Study Category
Investigator Initiated
  Payment Options: 
  * Option 1:  No protocol-required services will be billed to patients or third-party payers.
 Does Not Need MCA 
  * Option 2:  Protocol-required services (i.e., routine care services) will be billed to patients
 or third-party payers. Must Have MCA 
  * Option 3:  Study is initiated and federally funded by a Government Sponsored
 Cooperative Group who will only pay for services that are solely conducted for research
 purposes and other protocol-required services (i.e., routine care services) will be billed to
 patients or third-party payers. Billing Grid Only Required, NO MCA 
  * Option 4:  Study involves only data collection and has no protocol-required clinical
 services. Does Not Need MCA 
  * Option 5:  Study is not described in any of the above options. Please describe the study
 and specify whether External Sponsor (i.e., industry, government, or philanthropic source)
 and/or patient/third party payer will pay for protocol required services. MCA MAY Be
 Required 
  
 
Payment Option
Option 4
  
  No MCA is needed per option selected above. 
  
 
Page 27 of 28

IRB-19-01700
Initial Application
Matthew Levin
22. Attachments
Type
Name
Version Status
Filename
Uploaded
 Date
Data Collection Sheet Appendix 3  -
 DataElements.xlsx
1
New
DataElements.xlsx
03/04/2019
Other - Other IRB
 Correspondance
Appendix 2a -
RESCUE ME Opt-out
 Request Handling
 Procedure.docx
1
New
Appendix 2a -
RESCUE ME Opt-out
 Request Handling
 Procedure.pdf
03/05/2019
Other - Other IRB
 Correspondance
Appendix 2b -
 Specification for
 Opt-out Computer
 Program.docx
1
New
Appendix 2b -
 Specification for
 Opt-out Computer
 Program.docx
06/11/2019
Other - Other IRB
 Correspondance
Appendix 4a -
 Notification Process
 Primary.pdf
1
New
Appendix 4a -
 Notification Process
 Primary.pdf
03/05/2019
Other - Other IRB
 Correspondance
Appendix 4b -
 Notification Process
 RRT.pdf
1
New
Appendix 4b -
 Notification Process
 RRT.pdf
03/05/2019
Other - Other IRB
 Correspondance
Appendix 5 - HRP416
 Waiver of Consent
 Documentation
1
New
Appendix 5 - HRP416
 Waiver of Consent
 Documentation.pdf
04/25/2019
Other - Participant
 Educational
 Materials
Appendix 1a -
 Waiver-Consent
 Information Sheet-
 CASES.docx
1
New
Appendix 1a -
 Waiver-Consent
 Information Sheet-
 CASES.docx
06/11/2019
Appendix 1b -
 Waiver-Consent
 Information Sheet-
 CONTROLS.docx
1
New
Appendix 1b -
 Waiver-Consent
 Information Sheet-
 CONTROLS.docx
06/11/2019
Instruments
REDCap RRT Survey
 Placeholder.docx
1
New
REDCap RRT Survey
 Placeholder.docx
06/11/2019
 
Page 28 of 28

 
9 
Final Protocol 
The final protocol was the same as the original protocol. No changes were made to the 
protocol during the study period.  
 
 

 
10 
Original Statistical Analysis Plan 
As submitted to the IRB Mar-June 2019 
 
For all interim reports, final study outcome report, and publication analysis our 
research team plans to use an array of parametric and non-parametric statistical 
approaches to evaluate the significances of the primary and secondary endpoints. 
These would include, but not limited to binomial proportion tests, students t-tests, 
Mann-Whitney U test, Fisher's exact test, and ANOVA. Additionally, we plan to 
examine any potential significant difference in the demographics of the patients 
admitted to 10W and 10E when compared to 9W and 9E, significant differences will be 
incorporated into a multi-variant model such as a linear/logistic regression model to 
further elucidate factors that may have confounded the # of RRT evaluation and 
escalation events.  
 
 

Final Statistical Analysis Plan 
December 2021 
 
Data from the first hospitalization for each patient during the study period were included in the study. 
The analyses focused on patient level, unit level, and alert level, individually. All variables were 
summarized using the appropriate descriptive statistics. Continuous variables were presented as 
mean (standard deviation) or median [interquartile range], as appropriate. Two-sample t test was used 
to examine differences in means and Kruskal-Wallis test was used to assess differences in median 
between study arms. Categorical variables were presented as count (percentage). Chi-square test was 
used for categorical variables associated different study arms.  
 
Patient Level Analysis 
Study arm at admission was used to define the treatment group throughout the patient level analysis. 
The intervention group referred to the patients initially assigned to the MEWS intervention, and the 
control group referred to the patients initially assigned to the non-MEWS intervention. Propensity 
score modeling with inverse probability of treatment weight (IPTW) was used to account for 
differences in baseline characteristics for patients between two treatment groups. To reduce influence 
from extreme weights, we used stabilized weights, where the weight for a MEWS subject was the 
proportion of subjects receiving MEWS divided by the subject’s propensity score, and the weights for 
a non-MEWS subject was the proportion of subjects receiving non-MEWS divided by one minus of the 
respective propensity score. These weights were then incorporated into all subsequent analyses.  
 
IPTW Negative binomial regression models were used to model the count data during the entire 
hospital stay, including the total number of RRT alerts, escalations, blood pressure support, medical 
orders, laboratory orders, and all orders. Treatment effect is expressed in terms of the adjusted ratio 
of the two counts between the intervention group and the control group. IPTW Logistic regression 
models were used to model the binary outcomes, including death within 30 days after admission, 
death in hospital, ever being escalated, and whether any medical or laboratory orders were ordered. 
Treatment effect is expressed in terms of the adjusted odds ratio. IPTW Cox proportional hazard 
regression analysis was used to assess the association of study arm with hospital length of stay. The 
hazard ratio is used to describe the relative probability of hospital discharge at a given time. A hazard 
ratio >1 is associated with a shorter hospital stay. As a confirmatory analysis, we also performed the 
regular covariates-adjusted regression analysis using the same set of predictors used in the 
propensity score model.  
 
Alert and Unit Level Analysis 
At the alert level, the status of ICU escalation, whether a type of medical order or laboratory order is 
placed, and the total number of medical orders and laboratory orders following each alert were 
summarized. The descriptive summary of alert level analysis was also performed for the subset of RRT 
alerts and primary alerts. Confusion matrices were used to analyze the performance of RRT alert 
versus none alert and any alert (rrt alert or primary alert) versus none alert, individually, in association 
with actual escalation and actual escalation to ICU within MEWS arm where all patients received 
MEWS escalation monitoring and provider alerting. In the floor unit level analysis, escalation status, 
ICU status, alert levels, and the total number alerts were summarized. 

 
2 
Statistical Analysis Plan Summary of Changes  
 
Primary outcome 
Analysis was limited to first hospitalization. For patients who had crossover between intervention and 
control arms, treatment group was determined by initial study arm from the first unit admission. 
Inverse probability of treatment weight was chosen as the method to account for differences between 
treatment arms.   
 
 
Secondary outcomes 
The analysis of the secondary outcomes was modified as follows: 
Secondary 
Outcome 
Number 
Secondary Outcome 
Description 
Modification 
3 
Number of participants 
requiring respiratory 
support 
This outcome was removed due 
to too few events. Only 20 
respiratory therapy orders were 
placed across both intervention 
and control groups. 
 
4 
Number of cardiac arrest 
episodes 
This outcome was removed due 
to too few events. There were no 
episodes of cardiac arrest. 
5 
Mortality Rate 
This was further clarified to be 
combined in-hospital and 30-day 
mortality. 
6 
Notification Frequency 
This outcome was removed. As 
implemented, the number of 
alerts generated per patient was 
capped at 3 per unit admission in 
both arms. 
 
7 
Number of calls 
This outcome was removed. It 
was intended to measure the 
number of calls to RRT initiated 
by clinicians independent of the 
alert system. Low data quality in 
the separate RRT call tracking 
system made the data unusable. 
 
Ad-hoc outcomes 
The following ad-hoc outcomes were added: 
1. Likelihood of ICU escalation within 12 and 24 hours 
2. Likelihood of earlier hospital discharge