SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Models and Baselines. We compare the performance of the following baseline and models. 1) Rule-Based Method: For the rule-based approach, we used the implementation from Plesinger et al. [2015], a winning entry in the PhysioNet 2015 challenge for false arrhythmia alarm reduction. Plesinger et al. [2015] test each channel in the record for regular heart activity using the QRS detection and derived R-R information [Plesinger et al., 2015]. 2) MLP: We apply the multi-layer perceptron as a feature extractor of the input waveform and then use a dense layer to classify. 3) SAE (supervised autoencoder): We use autoencoder to learn a lower dimensional representation from 10-second waveform signals, and simultaneously minimize the reconstruction loss from autoencoder and the cross entropy loss as described in [Lehman et al., 2018]. One notable difference from [Lehman et al., 2018] is that we utilize the entire 10-second waveform segment as the model input, without heartbeat detection for learning beat-by-beat representations. 4) Transformers: We utilize the Transformer encoder [Vaswani et al., 2017] as the feature extractor of the input waveform. 5) CNN: We use a light-weight 1-D convolutional neural network as a feature extractor [Zhou et al., 2022]. 6) CNN+CL: We use a light-weight 1-D convolutional neural network as a feature extractor and augment it with contrastive learning [Zhou et al., 2022]. 7) FCN: We use a fully-connected convolutional network as the feature extractor of the input waveform [Zhou et al., 2022]. 8) FCN+CL: We use a fully-connected convolutional network as the feature extractor of the input waveform and augment it with contrastive learning [Zhou et al., 2022]. 9) BeatGAN: BeatGAN [Zhou et al., 2019] is an GAN-based unsupervised anomaly detection algorithm for time series data. 10) TanoGAN: TanoGAN [Bashar and Nayak, 2020] is a novel GAN-based unsupervised method for detecting anomalies when a small number of data points are available. 11) Diffusion+CL: we use a diffusion model to reconstruct trajectories of physiological signal and classify alarms based on distances between the reconstructed and the actual trajectories [Wu et al., 2023].
For the supervised approaches, including their variants with contrastive learning, we conducted model training for a maximum of 500 epochs. For the real-time events, hyperparameters were chosen through a grid search, and the Adam optimizer [Kingma and Ba, 2014] with a weight decay of 0.005 was employed to minimize both binary cross-entropy (BCE) loss and the discriminative constraint. Hyperparameter tuning was carried out, and a comprehensive description of the hyperparameter settings for all models can be found in the Appendix. For each machine learning model, we identified the hyperparameter setting with the best validation performance and proceeded to train the model with that setting using 10 different random seeds. We report the mean and standard deviation from the 5 random seeds with best validation performance. Model selection was based on the best Challenge Score on the validation set.
For retrospective events, we used the optimal hyper-parameter setting from the grid search of the real-time event, and proceeded to train the model with that setting using 10 different random seeds.
Table 4: Real-Time event performance from the VTaC database. Mean and standard deviation from 5 seeds shown. Top-performing models for each specific metric highlighted in bold. CL=Contrastive Learning.
Methods	TPR	TNR	PPV	F1	Score	AUC
Rule-based	0.942	0.629	0.502	0.655	67.32	-
Supervised	MLP	0.597±0.087	0.691±0.09	0.441±0.037	0.502±0.015	45.58±1.10	0.706±0.008
SAE	0.848±0.028	0.790±0.032	0.617±0.028	0.713±0.012	68.77±1.11	0.896±0.007
Transformer	0.837±0.039	0.707±0.060	0.535±0.048	0.651±0.030	62.73±2.78	0.852±0.006
CNN	0.928±0.006	0.782±0.019	0.629±0.019	0.750±0.013	76.17±1.20	0.936±0.009
FCN	0.920±0.025	0.855±0.018	0.717±0.024	0.805±0.016	80.08±2.46	0.949±0.006
Supervised+CL	CNN+CL	0.930±0.016	0.823±0.012	0.676±0.012	0.783±0.003	79.07±0.99	0.943±0.005
FCN+CL	0.931±0.028	0.811±0.046	0.666±0.048	0.775±0.024	78.41±0.87	0.932±0.008
Generative	BeatGAN	0.597±0.100	0.597±0.091	0.373±0.033	0.455±0.037	41.02±2.71	0.597±0.028
TAnoGAN	0.704±0.053	0.611±0.076	0.421±0.027	0.524±0.008	47.61±0.87	0.657±0.012
Diffusion+CL	0.853±0.054	0.517±0.040	0.412±0.013	0.555±0.016	52.51±2.61	0.685±0.017
Evaluation The evaluation metrics for false alarm reduction are true positive rate (TPR), true negative rate (TNR), positive predictive value (PPV), F1-score, and area under the ROC (receiver operating characteristic) curve (AUC). The PhysioNet Challenge 2015 also provides an official scoring mechanism for evaluating [Clifford et al., 2015a]. It is defined as score  $= (TP + TN) / (TP + TN + FP + 5 * FN)$ , where TP is true positives, FP is false positives, FN is false negatives, and TN is true negatives. The Challenge Score focuses more on the TPR value, since mistakenly classifying a true alarm as false results in significantly more severe consequences. The PPV, F1, TPR, TNR and Challenge Score are determined based on a threshold value of predicted probability that maximizes the validation Challenge Score as a cut-off value to determine true vs. false alarms in the test set.
6 Performance in Real-Time VT Alarm Classification
In this section, we present the experimental results of eleven models for the real-time event as outlined in Table 4. Results for the retrospective event are presented in the Appendix Table 8. The winning entry of the 2015 PhysioNet challenge using a Rule-based approach serves as a reference point for benchmarking, achieving a Challenge Score of 67.32 in the real-time event.
Our findings indicate that both FCN and CNN, along with their contrastive learning variants, demonstrated superior performance when compared to other models. Notably, FCN emerged as the top performer, achieving the highest Challenge Score (80.08±2.46) and AUC (0.949±0.006), surpassing other models. Contrastive learning enhanced the performance of 1-D CNN, as evidenced by a notable improvement in the Challenge Score from 76.17 to 79.07. However, this performance boost was not observed in the case of FCN.
Our study reveals distinctive findings when contrasted with Zhou et al. [2022]. In their investigation, the application of FCN to a considerably smaller dataset from the 2015 PhysioNet Challenge resulted in significantly inferior performance compared to the lightweight 1-D CNN and  $\mathrm{CNN + CL}$ . In contrast, our analysis, conducted on the VTaC dataset, demonstrated that FCN outperformed 1-D CNN in terms of AUCs and Challenge Scores. This difference in outcomes could potentially be attributed to the common observation that increasing the size of the labeled dataset often leads to improved performance in more complex models.
Generative approaches that previously demonstrated superior performance [Wu et al., 2023] using the MIMIC II annotated arrhythmia dataset [Saeed et al., 2011, Aboukhalil et al., 2008] faced challenges when applied to the VTaC dataset. In particular, the diffusion model with contrastive learning [Wu et al., 2023] outperformed other baselines when applied to the MIMIC dataset but under-performed with the current VTaC dataset. The performance gap can be attributed to several potential factors. Firstly, the imbalanced label distribution in VTaC, with a relatively lower rate of true alarms at  $28.6\%$ , in comparison to over  $50\%$  true alarms in the MIMIC II dataset, presents a potential limitation, particularly for generative approaches proposed in [Wu et al., 2023], which depended on training
with abundant samples of true positives. Secondly, less than  $40\%$  of alarm records in VTaC included arterial blood pressure (ABP) waveforms, in contrast to the MIMIC annotated arrhythmia dataset, which had a higher percentage of ABP records due to its specific sampling criteria. Finally, the MIMIC II arrhythmia dataset is a single-center, relatively homogeneous dataset collected from patient monitors of a single monitoring system, whereas VTaC is a multi-center dataset.
Previous algorithm development in false arrhythmia reduction has been hampered by the use of small, single-center, or relatively homogeneous datasets. This limitation hinders their generalizability and real-world applicability. The introduction of this large-scale annotated VT alarm dataset provides a valuable opportunity to address this challenge. The VTaC dataset presented in this study, encompasses data collected from ICUs in three major US hospitals and incorporates data from three major bedside monitor manufacturers. By encompassing data from diverse sources, this dataset enables the evaluation and refinement of algorithms in a broader context, across a more diverse monitoring devices and clinical settings. Furthermore, the random selection of VT alarm events from the available data for annotation ensures that the dataset's distribution and the proportion of true alarms closely resemble those observed in real ICU settings. This approach enhances the dataset's representativeness and allows for a more accurate evaluation of the models' performance in practical clinical scenarios.
Safety & Ethical Discussion We raise several safety and ethical considerations. Firstly, as the dataset contains waveform recordings from real patients, we acknowledge the importance of patient privacy and data protection. To ensure privacy, we have de-identified the data by removing any personally identifiable information. Secondly, the process of labeling the dataset involved clinical experts reviewing waveform recordings and making labeling decisions. We recognize the potential influence of individual subjectivity and variability among annotators. To mitigate this, we have sought to ensure robustness by obtaining labeling decisions from multiple experts for each alarm event. This approach helps to capture diverse perspectives and minimize potential bias in the annotations. We have made efforts to curate a diverse and representative dataset, but there might still be underlying biases that could impact the performance of the machine learning models.
Limitations & Future Work The newly introduced annotated VT alarm database is specifically designed to tackle the challenges related to reducing false VT alarms. The VT alarm events are randomly sampled, and thus the selected VT events cannot be viewed as a comprehensive collection of all VT alarm events for an individual patient. As such, the database is not designed for long-term forecasting of arrhythmia episode onset. Another limitation of our study is the absence of detailed clinical information accompanying the waveform recordings. We leave the collection of matched clinical data for future research endeavors. Finally, while the biases from expert annotators in this dataset is greatly reduced from having multiple annotators, we acknowledge that biases can still be present if two or more annotators have the same bias.
In our analyses, we have opted to construct our machine learning models by utilizing data from up to 10 seconds before the alarm onset (except the contrastive learning based approaches, which sample data from prior to 10 seconds before the alarm onset). Additionally, this study involved the inclusion of a maximum of four channels of available waveforms per event to formulate our models. For future investigations, we aim to explore models capable of more effectively handling higher-dimensional multi-channel waveform data encompassing longer sequences. This will enable us to fully leverage data from earlier time points and identify patterns across multiple channels of physiological waveforms.
Ventricular tachycardia is the most challenging arrhythmia to detect reliably, and remains a continuing problem in ICUs despite decades of effort from industrial and academic algorithm developers. We present a new annotated VT database to address the challenges associated with reducing false VT alarms. We conducted a comprehensive benchmarking of various machine learning models utilizing this annotated VT dataset. By providing this resource as an open-access database, freely available to the research community, we aim to foster collaboration, facilitate benchmarking, and encourages the advancement of research efforts in the field of arrhythmia alarm analysis.
This study was funded by NIH grant R01EB030362. L Lehman was in part funded by MIT-IBM Watson AI Lab. The authors would like to express their gratitude to Dr. Eric Gottlieb and Dr. Xavier Borrat Frigola for their contributions to the annotations. Additionally, we extend our appreciation to Shaherul Haque and Anthony Baez at MIT for their technical assistance.
Barbara J Drew, Patricia Harris, Jessica K Zegre-Hemsey, Tina Mammone, Daniel Schindler, Rebecca Salas-Boni, Yong Bai, Adelita Tinoco, Quan Ding, and Xiao Hu. Insights into the problem of alarm fatigue with physiologic monitor devices: a comprehensive observational study of consecutive intensive care unit patients. *PloS one*, 9(10):e110274, 2014.
Maria Cvach. Monitor alarm fatigue: an integrative review. Biomedical instrumentation & technology, 46(4):268-277, 2012.
Gari D Clifford, Ikaro Silva, Benjamin Moody, Qiao Li, Danesh Kella, Abdullah Shahin, Tristan Kooistra, Diane Perry, and Roger G Mark. The physionet/computing in cardiology challenge 2015: reducing false arrhythmia alarms in the icu. In 2015 Computing in Cardiology Conference (CinC), pages 273-276. IEEE, 2015a.
GD Clifford, I Silva, B Moody, Q Li, D Kella, A Chahin, Kooistra T, D Perry, and RG. Mark. False alarm reduction in critical care. Physiological Measurements, 37(8):E5-E23, 2016.
Anton Aboukhalil, Larry Nielsen, Mohammed Saeed, Roger G Mark, and Gari D Clifford. Reducing false alarm rates for critical arrhythmias using the arterial blood pressure waveform. Journal of biomedical informatics, 41(3):442-451, 2008.
Yuerong Zhou, Guoshuai Zhao, Jun Li, Gan Sun, Xueming Qian, Benjamin Moody, Roger G Mark, and Li-wei H. Lehman. A contrastive learning approach for ICU false arrhythmia alarm reduction. Nature Scientific Reports, 2022.
L Littmann, EG Olson, and MA Gibbs. Initial evaluation and management of wide-complex tachycardia: A simplified and practical approach. Am J Emerg Med, 37(7):1340-1345, 2019.
D. Kiyasseh, T. Zhu, and D. A. Clifton. Clocs: Contrastive learning of cardiac signals across space, time, and patients. In Proceedings of the 38th International Conference on Machine Learning, page 5606-5615, 2021a.
Filip Plesinger, Petr Klimes, Josef Halamek, and Pavel Jurak. False alarms in intensive care unit monitors: detection of life-threatening arrhythmias using elementary algebra, descriptive statistics and fuzzy logic. In 2015 Computing in Cardiology Conference (CinC), pages 281-284. IEEE, 2015.
GB Moody and RG Mark. The impact of the MIT-BIH arrhythmia database. IEEE Eng in Med and Biol, 20(3):45-50, 2001.
Gari Clifford, Ikaro Silva, Benjamin Moody, Qiao Li, Danesh Kella, Abdullah Shahin, Tristan Kooistra, Diane Perry, and Roger Mark. The physionet/computing in cardiology physionet challenge 2015: Reducing false arrhythmia alarms in the icu. In Proceedings of Computing in Cardiology, 2015b.
Gari D Clifford, Chengyu Liu, Benjamin Moody, H Lehman Li-wei, Ikaro Silva, Qiao Li, AE Johnson, and Roger G Mark. Af classification from a short single lead ecg recording: The physionet/computing in cardiology challenge 2017. In 2017 Computing in Cardiology (CinC), pages 1-4. IEEE, 2017.
MM Pelter, MG Carey, S Al-Zaiti, J Zegre-Hemsey, C Sommargren, L Isola, P Prasad, D Mortara, and F Badilini. An annotated ventricular tachycardia (vt) alarm database: Toward a uniform standard for optimizing automated vt identification in hospitalized patients. Ann Noninvasive Electrocardiol, 28(4), July 2023.
Appendix A Additional VT Alarm Dataset Description
This dataset is a compilation of patient waveforms sourced from multiple institutions, and the process of collecting this data has received approval from the respective Institutional Review Boards (IRBs) of each participating institution. Requirement for individual patient consent was waived because the project did not impact clinical care and all protected health information was de-identified. It is important to emphasize that the alarms and waveform records included in this dataset encompass data collected up until 2022 and do not overlap with those of the PhysioNet Challenge 2015.
Continuous multi-channel waveform recordings were collected from ICUs of three major US hospitals. The dataset consists of a total of 20,158 continuous waveform recordings collected between 2013 and 2022, and represents data collected from three major bedside monitor manufacturers. The average duration of the patient waveform recording is 4.5 days. We randomly extracted recordings with at least one VT alarm. The resulting dataset consists of a total of 18,465 waveform alarm events, sourced from a randomly selected 2,376 unique patient waveform recordings (with at least one VT alarm). The annotated VT alarm events (5,742 alarms from 2,376 patient records) were randomly drawn from this collection, with the constraint that no more than a maximum of five VT alarms can be sampled from the same patient waveform records to ensure diversity. After removing confirmed Reject/Uncertain events (by at least two independent annotators), 5,037 alarm events (from 2,260 unique patient records) remain for algorithm training and evaluation. Table 5 displays the number of events receiving unanimous vs conflicting decisions respectively. Unanimous category represents events with decisions from two independent annotators in agreement. Figure 2 illustrates the VTaC data collection and annotation process.
Table 5: Annotated VT alarm events categorized by annotation agreement.
Unanimous (79%)	Conflicts (21%)	Total
Adjudicated	Un-Adjudicated
Alarm Events	4,534	816	392	5,742
Labeling Decisions	4,534x2	816x3	392x2	12,300
Events with Final Decisions	4,534	816	0	5,350
Figure 2: VTaC data collection, event sampling and annotation pipeline.
ECG and pulsatile signals were recorded by the bedside monitor at various sampling frequencies ranging from  $100\mathrm{Hz}$  to  $250\mathrm{Hz}$ . ECG signals were recorded at a minimum of  $240\mathrm{Hz}$ . ABP and PPG were all sampled at the same rate which would be either  $100 / 125 / 240\mathrm{Hz}$ . All of the signals were resampled to  $250\mathrm{Hz}$  for consistency.
Among the annotated 5,037 VT events, all events have at least two channels of ECG, and  $36\%$  and  $91\%$  of those events also had ABP and PPG respectively. The median number of ECG leads per waveform recording is 3. In Table 6, we show the summary statistics of the number of ECG leads per waveform recording in the dataset.
Appendix B Annotation Tool and Team
Annotation was performed using an open-source annotation platform, PhysioTag, which enables experts to collaboratively annotate physiological waveform records using a standard web browser.
Table 6: Summary statistics for the number of ECG leads per waveform recording in the VTaC dataset.
# of ECG Leads
Average	3.4
Median	3
25% percentile	3
75% percentile	4
Table 7: Real-Time event: Test set performance of the model that exhibited the best validation performance among the 10 random seeds. Top-performing models for each specific metric highlighted in bold. CL=Contrastive Learning.
Methods	TPR	TNR	PPV	F1	Score	AUC
Rule-based	0.942	0.629	0.502	0.655	67.32	-
Supervised	MLP	0.708	0.562	0.391	0.503	45.33	0.703
SAE	0.847	0.803	0.630	0.723	69.43	0.905
Transformer	0.796	0.719	0.529	0.636	60.10	0.844
CNN	0.927	0.788	0.635	0.754	76.43	0.936
FCN	0.949	0.835	0.695	0.802	81.96	0.948
Supervised + CL	CNN+CL	0.912	0.835	0.687	0.784	77.92	0.946
FCN+CL	0.912	0.829	0.680	0.779	77.55	0.922
Generative	BeatGAN	0.803	0.473	0.377	0.513	46.27	0.638
TAenoGAN	0.723	0.580	0.406	0.520	47.16	0.651
Diffusion+CL	0.883	0.499	0.412	0.561	53.66	0.691
The software is freely available on PhysioNet (https://physionet.org/content/physiotag/). Events are assigned to annotators dynamically. When a new annotator joins the project, they are assigned 100 randomly-selected events that have not yet been annotated. After they have finished reviewing the assigned events, they can assign themselves a new batch of events. For our VT annotation task, users were given the options of "True" for when they believe the alarm was correct, "False" for when they believe the alarm was incorrect, "Uncertain" for when they are unsure which annotation to assign, "Reject" for when the alarm is unreadable due to noise, artifacts, or other hindrance, and "Save for Later" for when the user would like to return to annotate this event at another time. For more detailed description of the annotation platform for this project, please see [McCullum et al., 2023].
The annotation team consists of six annotators, including a leading arrhythmia analysis expert physician who built the MIT-BIH Arrhythmia database, and a highly-experienced board certified cardiac arrhythmia technician. The remaining three clinical annotators had the following expertise: an endocrinologist and internist with extensive experience in Emergency Medicine and cardiovascular intensive care; an inpatient physician, board certified in nephrology and internal medicine, with substantial experience managing patients in critical care and telemetry settings; and an anesthesiologist working in an ICU setting. The annotation team also included a senior biomedical research engineer with decades of experience in arrhythmia analyses.
Appendix C Additional Results on Model Performance
C.1 Real-time event
In this section, we present additional results for the real-time VT alarm classification task. Table 7 displays the test set performance of the model that exhibited the best validation performance among the 10 random seeds. Model selection was based on the best Challenge Score on the validation set. In Figure 3 we plot the ROC curves and their operating points for a subset of the models in the real-time event.
Figure 3: ROC curves and their operating points for a subset of models in the real-time event. Operating point (marked as dots) for each model on the ROC curve was selected using the threshold that resulted in the best validation Challenge Score.
Table 8: Retrospective event performance from the VTaC database. Mean and standard deviation from 5 seeds shown. Top-performing models for each specific metric highlighted in bold. CL=Contrastive Learning.
Methods	TPR	TNR	PPV	F1	Score	AUC
Rule-based	0.964	0.571	0.471	0.633	65.54	-
Supervised	MLP	0.585±0.064	0.711±0.162	0.445±0.168	0.505±0.034	46.02±2.85	0.695±0.016
SAE	0.836±0.046	0.761±0.053	0.585±0.043	0.687±0.022	66.01±2.37	0.878±0.0159
Transformer	0.802±0.075	0.677±0.066	0.500±0.031	0.613±0.007	58.20±2.07	0.811±0.029
CNN	0.923±0.020	0.803±0.025	0.651±0.026	0.763±0.015	76.94±1.61	0.942±0.003
FCN	0.923±0.012	0.786±0.025	0.632±0.025	0.750±0.016	75.84±1.86	0.925±0.014
Supervised+CL	CNN+CL	0.920±0.023	0.837±0.021	0.692±0.025	0.789±0.014	78.85±1.95	0.946±0.007
FCN+CL	0.921±0.013	0.836±0.011	0.691±0.013	0.789±0.001	78.95±1.41	0.933±0.007
Generative	BeatGan	0.781±0.093	0.497±0.114	0.385±0.023	0.513±0.013	46.22±1.77	0.639±0.017
TAnoGAN	0.749±0.051	0.546±0.056	0.397±0.014	0.518±0.004	46.97±0.49	0.648±0.005
Diffusion+CL	0.630±0.183	0.503±0.046	0.462±0.069	0.518±0.038	49.70±2.42	0.569±0.093
C.2 Performance in Retrospective Event
In the context of the retrospective event, machine learning models were assessed for their ability to reduce false arrhythmia alarms using not only the data preceding the alarm but also five seconds of data after the alarm. For retrospective events, we used the optimal hyper-parameter setting from the grid search of the real-time event, and trained the model with that setting using 10 different random seeds. Table 8 displays the performance averaged over the models from the 5 random seeds with the best validation performance.