SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Li-wei H. Lehman * MIT
Benjamin Moody MIT
Harsh Deep MIT
Feng Wu  
MIT
Hasan Saeed MIT
Lucas McCullum  
MIT
Diane Perry  
MIT
Tristan Struja  
MIT
Qiao Li
Emory University
Gari Clifford  
Emory University, Georgia Tech
Roger G. Mark  
MIT
False arrhythmia alarms in intensive care units (ICUs) are a continuing problem despite considerable effort from industrial and academic algorithm developers. Of all life-threatening arrhythmias, ventricular tachycardia (VT) stands out as the most challenging arrhythmia to detect reliably. We introduce a new annotated VT alarm database, VTaC (Ventricular Tachycardia annotated alarms from ICUs) consisting of over 5,000 waveform recordings with VT alarms triggered by bedside monitors in the ICUs. Each VT alarm in the dataset has been labeled by at least two independent human expert annotators. The dataset encompasses data collected from ICUs in three major US hospitals and includes data from three leading bedside monitor manufacturers, providing a diverse and representative collection of alarm waveform data. Each waveform recording comprises at least two electrocardiogram (ECG) leads and one or more pulsatile waveforms, such as photoplethysmogram (PPG or PLETH) and arterial blood pressure (ABP) waveforms. We demonstrate the utility of this new benchmark dataset for the task of false arrhythmia alarm reduction, and present performance of multiple machine learning approaches, including conventional supervised machine learning, deep learning, contrastive learning and generative approaches for the task of VT false alarm reduction.
Within intensive care units (ICUs), bedside monitors generate a substantial number of alarms, a considerable proportion of which are false [Drew et al., 2014]. These false alarms not only burden clinicians with heightened cognitive strain but also have the potential to obscure genuine alarms, thereby endangering patient safety. Arrhythmia alarms, accounting for  $45\%$  of overall ICU alarms, in particular, pose a significant challenge, contributing to alarm fatigue that increases the risk of healthcare providers potentially overlooking true life-threatening events [Drew et al., 2014, Cvach, 2012].
Accurate and reliable algorithms that can distinguish between true vs. false arrhythmia alarms can improve the overall effectiveness of monitoring systems in ICUs. However, the development and evaluation of such algorithms rely on the availability of high-quality and representative datasets with ground-truth labels from human experts.
*Corresponding author: lilehman@mit.edu.
We focus on the problem of detecting false alarms for one of the life-threatening arrhythmias, ventricular tachycardia (VT), defined as five or more ventricular beats with a heart rate higher than 100 bpm (beats-per-minute) [Clifford et al., 2015a, 2016]. Among all the life-threatening arrhythmia alarms, VT alarms are one of the most commonly occurring alarms [Drew et al., 2014, Aboukhalil et al., 2008] and false VT alarms have proven to be the most challenging to detect reliably [Aboukhalil et al., 2008, Clifford et al., 2015a, 2016, Zhou et al., 2022].
In this study, we present a new annotated VT database designed to address the challenges associated with false arrhythmia alarm reduction. The dataset consists of a total of over 12,000 labeling decisions from six experts who reviewed over 5,000 waveform recordings with VT alarms triggered by patient bed-side monitors in an ICU setting. These records were sourced from multiple ICUs from three major hospitals in the United States, providing a diverse and representative collection of waveform data. Each waveform recording comprises electrocardiogram (ECG) leads and one or more pulsatile waveforms, such as photoplethysmogram (PPG or PLETH) and arterial blood pressure (ABP) waveforms.
One important aspect of our dataset is the inclusion of labeling decisions from multiple independent human annotators. This approach helps to ensure the robustness of the dataset by reducing the potential impact of individual subjectivity and variability among annotators. The availability of such a comprehensive and annotated dataset enables researchers and practitioners to evaluate and compare the performance of various machine learning approaches for false arrhythmia alarm reduction. We demonstrate the utility of our benchmark dataset by evaluating multiple machine learning approaches, including conventional supervised machine learning, deep learning, contrastive learning, and generative approaches.
By addressing the challenges of false arrhythmia alarms and providing a benchmark dataset, this research contributes to the ongoing efforts to improve patient safety, reduce alarm fatigue, and enhance the efficiency of monitoring systems in ICUs. The introduced dataset VTaC, to the best of our knowledge, represents the largest open-access, multi-center VT alarm dataset, meticulously annotated by highly skilled clinical experts. It provides a valuable resource for developing algorithms to reduce false VT alarms, a longstanding and unsolved challenge in ICUs. Data will be available on PhysioNet at www.physionet.org/content/vtac/.
2 Background and Machine Learning Challenges
In this section, we outline challenges in developing machine learning approaches for false VT alarm reduction.
Artifact or noise. Erroneous triggers of VT alarms typically occur due to noise and ECG-related artifacts, such as electrode movements, poor electrode contact, or other technical reasons. Ventricular arrhythmias are characterized by distortion of beat morphology with a broader QRS complex. However, a challenge to be overcome is that noise and artifacts can exhibit morphologies similar to abnormal QRS complexes, almost indistinguishable from the periodic anomalies from a true VT episode [Clifford et al., 2015a].
Other rhythms with similar appearance. VT and other rhythms, such as atrial fibrillation (A-Fib) and aberrantly conducted supra ventricular tachycardia (SVT) may exhibit similar characteristics on an ECG, making it difficult to distinguish them [Littmann et al., 2019]. For example, both VT and SVT with aberrant conduction can have wide QRS complexes on the ECG. In some cases, patients may have multiple simultaneous rhythms, further complicating the interpretation. For instance, VT may coexist with A-Fib or SVT, making it challenging to identify and differentiate each rhythm component.
Individual differences and variable morphology. VT can have variable morphologies, which means it can appear differently in different patients or even within the same patient. This variability can make it challenging to identify VT solely based on ECG characteristics immediately prior to the alarm onset. Inspection of the same individual patient's prior ECG characteristics at baseline may be necessary.
Long sequence with complex patterns and Missing Channels. In the false alarm reduction problem, one major challenge stems from the long time-series sequence and the sparse availability of labeled data [Clifford et al., 2015a, Kiyasseh et al., 2021a, Zhou et al., 2022]. At a sampling frequency of
$250\mathrm{Hz}$ , a 5-minute segment of a single channel of ECG contains 75000 samples. The label of each record, on the other hand, is just TRUE and FALSE, providing limited supervised information to train a deep learning model. Additionally, missing channels pose a significant challenge for false alarm reduction.
Sparse availability of high-quality labeled data. While machine learning and deep learning have made significant advances in many domains, including image and voice analyses, the application of deep learning in physiological waveform analysis has had limited success, partly due to limited availability of high-quality labeled data. The best performing approaches for false arrhythmia alarm detection in the 2015 PhysioNet Challenge rely on a combination of expert-defined rule-based logic analysis [Plesinger et al., 2015] and simple machine-learning models, while, in comparison, conventional deep learning approaches generally under-performed in the 2015 PhysioNet Challenge [Clifford et al., 2015a].
3.1 Related Works in Annotated Arrhythmia Datasets
Several efforts have been made in the development and curation of annotated arrhythmia datasets, which serve as valuable resources for the development and evaluation of algorithms aimed at arrhythmia detection and classification. These datasets contribute to advancing the field of cardiac monitoring and have facilitated the progress of machine learning techniques in arrhythmia analysis. In this section, we highlight some notable related works in the domain of annotated arrhythmia datasets.
One seminal dataset widely used in arrhythmia research is the MIT-BIH Arrhythmia Database [Moody and Mark, 2001]. It comprises 48 ECG records (each slightly more than 30 minutes) obtained from long-term Holter recordings. These records include a mix of randomly selected and specifically chosen cases, showcasing a broad spectrum of typical clinical waveforms, artifacts, and complex arrhythmias, both ventricular and supraventricular. The dataset has been instrumental in benchmarking different algorithms for arrhythmia classification and contributed to the evaluation of signal processing and machine learning techniques. The MIT-BIH Arrhythmia Database does not include annotations for arrhythmia alarms generated by commercial bed-side monitors.
The 2015 PhysioNet Challenge event [Clifford et al., 2015b, 2016] focused on five types of life-threatening arrhythmias, including ventricular tachycardia, asystole, extreme bradycardia, extreme tachycardia, and ventricular fibrillation/flutter. The goal of the Challenge was to reduce the number of false alarms, while avoiding suppression of true alarms. The Challenge consists of two events: (1) real-time classification using only data up to the alarm onset; (2) retrospective analysis in which the contestants are allowed to use the 10-second data after the alarm onset for classification.
The 2017 PhysioNet/CinC Challenge [Clifford et al., 2017] is a collection of single-lead ECG recordings (each between 30 to 60 seconds) from wearable devices. The primary objective of this dataset is to facilitate the development of algorithms for the classification of cardiac rhythms into four main categories: normal sinus rhythm, atrial fibrillation (AF), an alternative rhythm, or noisy recordings that cannot be classified.
Most recently, [Pelter et al., 2023] introduced the UCSF VT dataset comprising 18,455 annotated VT alarms from 858 ICU patients at UCSF. Their research differs from ours in several key aspects. The UCSF VT dataset was exclusively collected from a single hospital, utilizing patient monitors from a single commercial bed-side monitoring vendor. In contrast, VTaC comprises a multi-center collection, incorporating data from monitors manufactured by three major vendors of commercial monitoring systems. As a result, VTaC provides a more diverse and comprehensive representation, which is crucial for the development of VT false alarm reduction algorithms. Furthermore, the UCSF VT dataset, as indicated by [Pelter et al., 2023], is not shared or made publicly available to the research community. Conversely, VTaC is an open-access resource, freely accessible to the research community.
3.2 Machine Learning Approaches to False Arrhythmia Alarm Reduction
Multiple algorithms and machine learning approaches have been proposed for arrhythmia analyses and false alarm reduction. Conventional algorithms [Aboukhalil et al., 2008, Li and Clifford, 2012]
and the best performing entries in the 2015 PhysioNet Challenge for false arrhythmia alarm reduction [F Plesinger et al., 2016, Kalidas and Tamil, 2015, Clifford et al., 2015a] rely on signal processing and expert-defined rule-based logic to analyze physiological signals, or using manually engineered features as inputs for machine learning classifiers. Due to the sparse availability of labeled data, much of the recent machine learning approaches for false alarm reduction focus on techniques that enable label-efficient learning, using e.g. multi-task learning [Schwab et al., 2018], supervised representation learning [Lehman et al., 2018], and self-supervised or contrastive learning [Kiyasseh et al., 2021b, Zhou et al., 2022]. Lehman et al. [2018] focus on VT false alarm detection, and use FFT-transformation of individual ECG beats for scalable learning. Kiyasseh et al. propose a family of self-supervised pretraining mechanisms based on contrastive learning for physiological signals[Kiyasseh et al., 2021b]. Zhou et al. [2022] proposed a contrastive learning framework to reduce five types of life-threatening arrhythmia alarms. More recently, Wu et al. [2023] proposed a conditional generative modeling approach for the task of VT false alarm reduction, and showed promising results from a diffusion contrastive learning model using an annotated arrhythmia dataset [Aboukhalil et al., 2008] from the MIMIC II database [Saeed et al., 2011].
4 Data Collection and Annotation Methodology
In this section, we briefly outline our data collection methodology and the annotation process. For detailed description, please refer to the Appendix.
We extracted and compiled a total of 18,465 waveform VT alarm events, corresponding to 2,376 unique patient waveform records from bed-side monitors from three leading commercial vendors. These records were sourced from multiple ICUs from three major hospitals in the United States, providing a diverse and representative collection of waveform data. Each waveform recording in our dataset consists of a 10-minute segment that encompasses the onset of the ventricular tachycardia (VT) alarm. This segment includes 5 minutes of waveform data preceding the alarm onset and 5 minutes following it. To maintain diversity, we randomly selected a maximum of five alarm events from any particular patient waveform record, yielding a total of 5,742 events for annotation. This ensures that a reasonable number of events were sampled from different patient records, preventing an over-representation of events from any single patient record. Waveform records were de-identified to ensure the anonymization of all identifiable information, including patient names, dates, and medical record numbers. All signals were resampled to  $250\mathrm{Hz}$ , and all signal labels were adjusted to match the nomenclature used in the PhysioNet Challenge 2015 database.
4.2 Annotation Process
Following the PhysioNet Challenge 2015, a VT episode is defined as five or more consecutive ventricular beats with heart rate higher than 100 beats-per-minute (bpm) [Clifford et al., 2015a]. Each event was reviewed and labeled by at least two annotators independently. For our task, annotators were given the options of "True" for when they believe the alarm was correct, "False" for when they believe the alarm was incorrect, "Uncertain" for when they were unsure which annotation to assign, "Reject" for when the alarm was unreadable due to noise, artifacts, or other reasons. In order to reconcile conflicts between two annotator decisions, an adjudication process was implemented to resolve the conflicts. These disagreements were resolved either through direct one-on-one discussions between the annotators involved or by an adjudicator's vote to break the tie.
A total of 5,742 events were annotated by at least two independent annotators. Two independent annotators reached unanimous decisions on 4,534 (78.96%) events, whereas 21.04% (N=1,208) of the events received conflicting labeling decisions by two human annotators. Among the events with conflicting decisions, 816 (66.55%) were adjudicated. After removing 392 un- adjudicated events, a total of 5,350 alarm events received final labeling decisions. See Table 5 in Appendix for the breakdown of the 5,742 events based on the annotation decisions.
Data Composition by Labeling Decisions Table 1 lists a summary of alarms categorized by annotation decisions among the 5,350 alarms with final labeling decisions. A total of 102 (1.91%) and 211 (3.94%) events received Uncertain and Reject decisions respectively. After excluding "Rejected"
Table 1: Composition of the alarms categorized by labeling decisions.
Unanimous	Adjudicated	Total
Total Alarm Events	4,534	816	5,350
Uncertain	19	83	102
Reject	120	91	211
True (T)	1,222	219	1,441
False (F)	3,173	423	3,596
Final (T/F) Events Included	4,395	642	5,037
Percent True Alarms	27.80%	34.11%	28.61%
and "Uncertain" events, the final dataset used for modeling contains 5,037 events, among which 4,395 events had unanimous annotation decisions, and the final labeling decisions of the 642 adjudicated events were based on the final decisions from the adjudicators. Figure 2 in the Appendix A illustrates the VTaC data collection and event sampling process.
Annotation Team The annotation team consists of six annotators, including a highly-experienced board certified cardiac arrhythmia technician, and a leading arrhythmia analysis expert physician who built the MIT-BIH Arrhythmia database. The team also includes three clinicians, and one biomedical signal processing engineer specializing in arrhythmia. We describe below expertise of three of the annotators who had contributed the most number of annotations. Annotator 1: Board certified cardiac arrhythmia technician with over 40 years of experience in interpreting and annotating arrhythmias from ECG recordings. Annotator 2: Developed the MIT-BIH Arrhythmia database; physician in internal medicine with decades of experience in real-time arrhythmia analyses and clinical interpretation of Holter recordings. Annotator 3: Endocrinologist and Internist with extensive experience in Emergency Medicine and cardiovascular intensive care. Additional details of the annotation team can be found in the Appendix.
Example True vs False VT Alarms in VTaC In Figure 1, we show waveform plots for two example VT alarms that have been labeled as a true vs. a false alarm respectively. In each figure, waveform recordings from the final 10-seconds prior to the VT alarm onset were shown.
(a) True VT alarm
(b) False VT alarm
Figure 1: Example true vs. false VT alarms. Each plot shows data in the 10-second interval immediately prior to the VT alarm onset. The alarm onset is marked with a vertical red-line at time 0. Figure (b) shows an example false VT alarm – the event corresponds to an episode of atrial fibrillation with rate-related aberration instead of a ventricular tachycardia.
4.3 Comparison with Other Annotated VT Databases
In Table 2, we compare our newly developed VT alarm dataset with existing annotated VT databases on PhysioNet, including the training and hidden test sets from the PhysioNet Challenge 2015 [Clifford et al., 2015a], and an annotated arrhythmia alarm dataset from MIMIC II [Saeed et al., 2011, Goldberger et al., 2000] as reported in Aboukhalil et al. [2008]. We compare these datasets in
terms of annotated alarm events, the percentage of true alarms, as well as percentage of alarm events with ECG, ABP, and PLETH. We note that VTaC is a federated database sourced from waveform recordings collected from multiple commercial bed-side monitors (as such, multi-vendor) across multiple ICUs in three US hospitals. We also note that the current dataset provides fine-grained pre-adjudication labeling decisions from human experts, which are not included in prior datasets. As we randomly select VT alarm events from available recordings in constructing our dataset, the federated VT alarm dataset contains a false alarm rate that closely reflects the actual false alarm rate in a real-world ICU setting [Drew et al., 2004].
Table 2: Comparison of Annotated VT Databases on PhysioNet. Pre-Adj Decisions indicate whether labeling decisions before the adjudication processes are available.
Challenge'15 Train	Challenge'15 Test	MIMIC II	VTaC
N (# VT alarms)	341	221	1,900	5,037
% True	26.67%	20.36%	53.40%	28.61%
Open Access*	Y	N	Y	Y
Multi-Vendor	Y	Y	N	Y
Pre-Adj. Decisions	N	N	N	Y
ECG (% events)	100%	100%	100%	100%
ABP (% events)	54%	57%	100%	36%
PLETH (% events)	83%	81%	0%	91%
5 Models & Evaluation
We demonstrate the utility of this new benchmark dataset for the task of false VT alarm reduction, and present performance of multiple machine learning approaches in both real-time and retrospective settings following PhysioNet Challenge 2015 [Clifford et al., 2015a]. In the real-time setting, the goal is to reduce false alarms in "real-time" without using information after the alarm onset. In the retrospective setting, algorithms are allowed to utilize data within a brief time interval following the alarm, such as the 30-second interval specified in the PhysioNet Challenge 2015. However, in the context of this study, the retrospective setting incorporates data up to 5 seconds after the alarm.
The final dataset used for modeling consists of 5,037 annotated VT alarms. The channels used for modeling included ECG, ABP, and PLETH (or PPG). We employed an 80-10-10 split for the train, validation, and test sets. The splits were performed at the patient record level rather than the individual alarm events level to ensure that events belonging to the same patient record are not distributed across the train, validation, and test sets. Table 3 displays the distribution of true alarms and signal types across the train, validation, and test set.
For the real-time event, 10-second window of waveform data immediately prior to the alarm onset were used as input to all of our models, with the exception that the generative models used a smaller window of data due to computational constraints. The retrospective events used 15-second window of waveform data to include the additional 5-second window of data immediately after the alarm onset. For each VT event, the model was fed with a total of four channels of waveform data, comprising two ECG leads, one ABP, and one PPG, all at a frequency of  $250\mathrm{Hz}$ .
Data Preprocessing For ECG, we perform the following filtering: 1) a high-pass filter with  $1\mathrm{-Hz}$  cutoff frequency to suppress residual baseline wander; 2) a second-order  $30\mathrm{Hz}$  Butterworth low-pass filter to reduce high frequency noise; and 3) a notch filter to eliminate power line interference. For PPG signal, we utilize a high-pass filter with a stopband frequency of  $0.3\mathrm{Hz}$  and a passband frequency of  $0.5\mathrm{Hz}$ , along with a low-pass filter with a passband frequency of  $5\mathrm{Hz}$  and a stopband frequency of  $8\mathrm{Hz}$ . We conduct z-normalization for all signals by utilizing the mean and standard deviation of each individual signal segment before feeding them into the model. In cases where signals are missing, we impute them with a value of 0.
Table 3: Composition of the train, validation and test splits. Unanimous decisions had two annotators who were both in agreement.
Overall	Train	Validation	Test
Alarms (count)	5,037	4,060	495	482
Records (count)	2,260	1,808	226	226
True Alarms (count)	1,441	1,163	141	137
% True	28.61%	28.65%	28.90%	28.48%
ECG (% events)	100%	100%	100%	100%
ABP (% events)	35.7%	35.6%	36.5%	35.6%
PLETH (% events)	90.6%	90.42%	89.83%	92.32%
Adjudicated	642	542	46	54
Unanimous	4,395	3,518	449	438
5.2 Models