ECG Representation Learning with Multi-Modal EHR Data
Authors: Sravan Kumar Lalam
Hari Krishna Kunderu
Shayan Ghosh
Harish Kumar A
Samir Awasthi
Ashim Prasad
Francisco Lopez-Jimenez
Zachi I Attia
Samuel Asirvatham
Paul Friedman
Rakesh Barve
Melwin Babu
Venue: Accepted by TMLR ()

Abstract: Electronic Health Records (EHRs) provide a rich source of medical information across different modalities such as electrocardiograms (ECG), structured EHRs (sEHR), and unstructured EHRs (text). Inspired by the fact that many cardiac and non-cardiac diseases influence the behavior of the ECG, we leverage structured EHRs and unstructured EHRs from multiple sources by pairing with ECGs and propose a set of three new multi-modal contrastive learning models that combine ECG, sEHR, and text modalities. The performance of these models is compared against different baseline models such as supervised learning models trained from scratch with random weights initialization, and self-supervised learning models trained only on ECGs. We pre-train the models on a large proprietary dataset of about 9 $million$ ECGs from around 2.4 $million$ patients and evaluate the pre-trained models on various downstream tasks such as classification, zero-shot retrieval, and out-of-distribution detection involving the prediction of various heart conditions using ECG waveforms as input, and demonstrate that the models presented in this work show significant improvements compared to all baseline modes.

Published in Transactions on Machine Learning Research (11/2023)
ECG Representation Learning with Multi-Modal EHR Data
Sravan Kumar Lalam 1,1
Hari Krishna Kunderu'
Shayan Ghosh
Harish Kumar A
Ashim Prasad'
Francisco Lopez-Jimenez"
Samir Awasthi'.2
Zachi Attia
Samuel J. Asirvatham
Paul A. Friedman
Rakesh Barve
Melwin Ba
1 Nference Inc.
Anamana Inc.
Mayo Clinic, USA
#Corresponding authors: (eravankumar www.mmrer..nffeeeeeee.em net, melwin@nference. .nt]
Reviewed OpenReview: kttps://openreet www.:eenntfftutee id= UXmvCwuTMG
Abstract
Electronic Health Records (EHRs) provide a rich source of medical information across dif- ferent modalities such as rocarrooads (ECG), structured EHRs (sEHR), and unstruc- tured EHRs (text). Inspired by the fact that many cardiac and non-cardiac diseases influence the behavior of the ECG, we leverage structured EHRs and unstructured EHRs from mul- tiple sources by pairing with ECGs and propose set of three new multi-modal contrastive learning models that combine FCG,, sEHR, and text modalities. The performance of these models is compared against different baseline models such as supervised learning models trained from scratch with random weights initialization, and self-supervised learning mod- els trained only on ECGs. We pre-train the models on a large proprietary dataset of about 9 million ECGs from around 2.4 million patients and evaluate the pre-trained models on various downstream tasks such classification, zero-shot retrieval, and out-of-distribution detection involving the prediction of various heart conditions using ECG waveforms as in- put, and demonstrate that the models presented in this work show significant improvements compared to all bascline modes.
1 Introduction
Electronic Hcalth Records (EHRs) are generated for every patient encounter or event and are becoming increasingly available in recent years. These are multi-modal in nature and capture rich phenotypic infor- mation of the patients over time in the form of structured EHRs and unstructured EHRs. Structured EHRs (denoted sEHR) contain information about diagnoses, procedures, medication prescriptions, lab tests, vitals, and more, while unstructured EHRs encompass clinical notes, radiology images, pathology images, echocar- diogram videos, time series ECG signals, etc. Recently multi-modal contrastive learning met applied to radiology and pathology images by pairing with the corresponding medical reports to learn medical image
representations (Zhang et al. 2022: Huang et al.) 2021 Boecking et al. 2022 Bannur et al. 2023 Lu et al. 2023) have shown promising results on downstream tasks such as classification, image-text retrieval, etc. These methods generally have two stages: (i) In stage I, the model pre-trained on large unlabelled data learn generic representations by maximizing the alignment between embeddings of different modalities in latent space. (ii) In stage II, the model is fine-tuned on task-specific labeled dataset by transferring the knowledge from the pre-trained model. However, ECG representation learning by pairing with EHRs via multi-modal contrastive learning is less explored. Uni-modal contrastive learning similar to Chen et al. (2020a) has been applied to the ECG domain to learn ECG representations (Kiyasseh et al. 2021 Diamant et al. 2022 Gopal et al.) 2021 Mehari Strodthoff 2022 Oh et al., 2022), but they lack the ability to compare different modalities latent space using similarity metrics like cosine similarity for use in zcro-shot transfer learning. Also, contrastive learning using multi-modal data produces high-quality representations they exploit information from multiple sources and extract semantics by aligning with various modalities.

---

Published in Transactions on Machine Learning Research (11/2023)
ECG is simple, non-invasive test that records the electrical activity of the heart and is helpful in diagnosing heart conditions and patient monitoring. In recent years, deep learning techniques have been employed on ECG data to predict various heart conditions, even in cases where diagnostic criteria using ECGs have
not been firmly established in clinical practice (Tison et al.)
2019
Hannun ct al.) 2019
Galloway et al.
2019
Attia et al. 2019a C Ko et al., 2020 Adedinsewo et al. 2020;
Christopoulos et al. 2020; Yao et al.
2021 Siontis et al. 2021 Cohen-Shelly et al. 2021 Bos et al., 2021 Grogan et al. 2021 Ahn al. 2022)
Gopal al. 2021 discusses that supervised learning models of this nature demand extensive, high-quality
datasets with precise annotations to achieve robust generalization real-world data. Unfortunately, within the healthcare domain, acquiring such labeled datasets is challenging, as they are scarce, expensive, and time-consumingg to obtain due to the necessity of trained physicians for the annotations. Motivated by the
following facts: (i) multi-modal contrastive learning applied to the general domain images (Radford et al. 2021 Jia et al. 2021 Goel et al.; 2022) as well as the medical domain images "Zhang et al. 2022 Huang
ct al. 2021 Boccking et al.) 2022 Bannur et al.] 2203: Lu al.) 2023) by pairing images with text data has demonstrated promising results; (ii) ECG signals contain information related to both cardiovascular and conccardioaa diseases (Venn et al. 2022;; (iit) EHR data capture rich phenotypic information of the patients over time, we address the challenges described previously by leveraging EHRs. We pair structured EHR and unstructured EHR data with ECGs to learn ECG reprosentations via multi-modal contrastive learning. In particular, we utilize diagnosis codes, procedure codes, and medication prescriptions from the structured EHR category and text data from various sources such as ECG reports, ECHO reports, radiology reports, pathology reports, microbiology reports, clinical notes, and surgical notes from the unstructured EHR category. Our contributions are summarised as follows:
We propose SEHR-BERT, BERT model pre-trained to encode sEHR modality for use in multi- modal contrastive learning models.
2. We propose a set of three multi-modal contrastive learning models that combine sEHR, ECG, and text modalities to learn ECG representations:
ECG-sEHR: A model that combines ECG and sEHR modalities,
ECG-Text: A model that combines ECG and text modalities,
sEHR-ECG-Text: A model that combines sEHR, ECG, and text modalities.
We then compare the effectiveness of the pre-trained models on downstream tasks such as linear classification, fine-tuning, zero-shot retrieval, and out-of-distribution detection with different baseline models including supervised learning models trained from scratch with random initialization and current -tate-of-the-art (SOTA) ECG-only self-supervised learning models.
2
Related Work
2.1 Contrastive Learning for General Domain Images
Self-supervised learning (SSL) using contrastive learning methods has emerged as a powerful pre-training technique to learn generic representations of the data. These methods learn representations either (i) by pulling the embeddings of similar pairs (positive pairs) together and pushing the embeddings of dissimilar pairs (negative pairs) apart in the latent embedding space (ii) by contrasting cluster assignments. Some of the notable works in computer vision include InfoNCE (Oord et al. 2018). SimCLR (Chen et al. (020a) SimCLRv2 (Chen et al. (020b). MoCo (He et al. 2020). SupCon (Khosla et al.) 2020), SEER (Goyal et al. 2021,, PIRL Misra & Maaten 2020), SwAV (Caron el al. 2020), and PCL (Li et al. 2021). These hods come under the category of uni- modal contrastive learning as they utilize only one type of data modality, i.e.,
images. Multi-modal contrastive learning by pairing general domain images with the corresponding image
captions to learn image-text embeddings jointly in the shared space (Radford al. 2021 Jia al. 2021
Goel et. al. 2022) has shown impressive results downstream tasks such as zero- feero//Twwshot learning.
2

---

Published in Transactions on Machine Learning Research (11/2023)
2.2
Contrastive Learning for Medical Domain Images
Uni-modal contrastive learning based on SimCLR has been applied to medical domain images (Aiizi el al. 2021 2022 Ciga et al. 2022 Wang et al. 2022 Srinidhi & Martel, 2021] Sowrirajan et al. 2021) to learn medical image representations. Motivated by some of the initial works in uni-modal contrastive learning, ConVIRT (Zhang et al.
2022 proposed a multi-modal contrastive learning method by pairing chest radiology images with the corresponding radiology reports. |Huang et al. (2021) extended ConVIRT for learning local and global representations by contrasting image sub-regions with words in the medical report. [Boecking al.) (2022; Bannur et al. (2023) made improvements in the radiology domain by leveraging longitudinal medical images, building a domain-specific language model for radiology reports, and adding Masked Language Modcling (MLM) loss to contrastive loss during joint vision-language pro-training. [u et al. (2023) applied multi-modal contrastive learning by pairing histopathology whole slide images with pathology reports. Our multi-modal contrastive learning models are largely inspired by COnVIRT (Zhang et al. 2022
2.3 Contrastive Learning for Time Series ECG signals
SimCLR and the other aforementioned uni-modal contrastive learning models were developed for use in computer vision. However, they have been adopted for use with time series ECG signals in subsequent works (Kiyassch ct al., 2021 Diamant et al. 2022; Gopal et al.] 2021 Mchari & Strodthoff] 2022 Oh al., 2022). The principle difference between CLOCS (Kiyasseh et al. 2021p. PCLR (Diamant et al. 2022) and
the 3KG (Gopal et al. 2021) models is in the way the positive pairs created. CLOCS treats consccutive non-overlapping segments and/or leads of the same ECG positive pairs. PCLR treats two ECGs of the same patient as positive pairs. 3KG constructs positive pairs by applying spatial augmentations such as rotation and scaling in vectorcardiogram (VCG) space after converting ECG to VCG, followed by temporal augmentations such as time masking in ECG space after converting VCG back to ECG. Cheng ct al. (2020) introduced adversarial Training to address intersubject variability while learning ECG representations using contrastive learning. MMehari & Strodthoti (2022) adapted SimCLR (Chen et al., 2020a), CPC (Oord et al, 2018), and SwAV (Caron et al., 2020) to ECG domain to learn ECG representations. Recently, Oh et al.
22022] proposed a pre-training method that combines CMSC from Kiyasseh et al. 20021 and Wave2 2.0 Baevski et al. 2020 from speech domain to learn local and global contextual ECG representations.
To the best of our knowledge, there is only one work that combincs ECGs with other modalities. Raghu et al. (2022) developed a SimCLR-like contrastive learning model that was pre-trained using multi-modal clinical time series data such as ECG signals and structared time series data (labs and vitals). The model utilizes 18-dimensional structured time series data from metabolic panel, blood pressures, heart rate, and SpO2. The model is shown to have achieved improved comparable performance over training from scratch on two downstream tasks: (i) Elevated mPAP and (ii) 24-hout mortality rate. To the best of our knowledge, we are the first to fully utilize the large landscape of electronic health records to learn ECG reprosentations.
3 Methods
3.1 SEHR-BERT: Structured EHR Model Pre-training
Several methods have been proposed to model structured EHRs based on BERT (Devlin et al.] 2018 BEHRT Li et al. 2020 BERT (Rasmy et al., 2021), CEHR-BERT (Pang et al. 2021). and CEHR- GAN-BERT (Poulain et al.] 2022). However, none of these pre-trained models are publicly available to use in our work. Moreover, the vocabulary in our dataset may not be aligned with the vocabulary of the mentioned models. So we developed sEHR-BERT. model pre-trained to encode sEHR data and produce sEHR representations based the BERT architecture (Devlin et al. 2018). We used vocabulary of size 28593, constructed from ICD diagnosis codes, ICD procedure codes, and medication prescriptions. These are collectively referred to as "medical codes" in this work. The input to the model is a sequence of medical codes sorted in ascending order based on medical codes' timestamps. Each code is processed by adding its corresponding medical code embodding, time embodding, and medical code type embodding and sent to the transformer encoder. Time embeddings are constructed so that codes falling within non-overlapping 7-day windows share common embedding. Medical code type embeddings are divided into different categories
3

---

Published in Transactions on Machine Learning Research (11/2023)
(i.e., discases, symptoms, procedures, special tokens, etc.). We used a custom BERT model with the number of layers, hidden size, and number of self-attention heads set to 5, 320, and 5 respectively. This model has 15M parameters. We initialize the model weights randomly and follow the BERT (Devlin et al. 2018) pre- training strategy, i.c., Maskod Language Modeling (MLM) to learn the representations of the structured EHR sequences. We minimize the MLM loss given by == K Lo=logp logp of the model, D = {Do Di,..., } is the sequence of medical codes of length N, M = {mo, m.,...,, mx} are indices of masked medical codes, and Dxr denotes the set unmasked medical codes. During training, the medical codes are masked with a probability of 15%, and the model is trained with AdamW [Loshchilov & Hutter 2019) optimizer and batch size of 512 for 100 epochs. We set an initial learning rate of 5c-4 and the learning rate is reduced by factor of 2 if the validation loss stops decreasing continuously for epochs.
D xy: θ), where θ are parameters
3.2
SEHR-ECG-Text. Joint SEHR, ECG and Text Pre-training
In this section, we describe the pre-training of the sEHR-ECG-Teex model, where we pair the ECG modality with sEHR and text modalities lo jointly learn multi-modal representations.
3.2.1 Preliminaries
MultiModal Versatile Networks (MMV) (Alayrac et al. 2020) apply contrastive learning to multi-modal data including video, audio, and text under the assumption that the video and audio modalities are more granular than the text modality. MMV discusses that applying contrastive loss in shared space (where all modalities are embedded into single shared vector space) may not maintain specificities, as implicitly assumes that all modalitics have equal granularity. To address this, MMV proposes to learn two separate embedding spaces: a fine-grained space where video and audio are matched and a coarse-grained space where text is matched with video and audio domains. This method is referred to as fine and coarse spaces (FAC), We hypothesize that sEHR, ECG, and text modalities do not exhibit equal granularity. Moreover, the ECGs are paired with sEHR and text data within specific time window surrounding the ECG acquisition timestamp, and Lokens are trimmed based on the input length accepled by the corresponding encoders, as we describe in Sections 4.2.2 and 4.2.3 This implies that the same level of information might not maintained between sEHR and text, so we adopt the FAC framework from MMV in our sEHR-ECG- ECG-Text model. We describe the methodology in detail in the following sections.
3.2.2 Notation
Let x E X be an instance defined by an instantiation of different modalities M : = {EmJ, m E M. In this study, we employ three modalitics: ECG xa E Xe, sEHR x, E X,, and text I2 E X, Specifically, Es, Te, and Hy represent the sequence of sEHR codes, ECG waveform samples, and sequence of text lokens respectively. Let Em : Xm Id be a parameterized modality-speeifii encoder that takes as input an instance I' from modality and produces modality-speeiffe reprosentation of dimension dm- These mopality-speeific reprosentations embedded into shared space Ω2 Rd, where represents the list of modalities that we embed into this space. For instance, = to denote joint ECG-sEHR space Nes: = el to denote joint ECG-Text space Pet, or = set to denote joint sEHR-ECG-Tew space Qeet- In this shared space, we maximize or minimize the alignment between different modalitics using the contrastive loss objective. The projection head Pmo+== Rdmm + Rdd is used to embed modality speciffi-reprreentatiin U. = Em (mm) into the shared space Ω2,, and we denote the resulting vector as Um, = P Pm--=((Ap)) (Fm (xm)), which signifies the representation of the input modality 2m in the shared space Qg.
3.2.3 Data Encoding
To oblain the modality-specifie representations, we use a convolutional neural network (CNN) customized to one dimension (E.) for the ECG modality, the pre-trained SEHR-BERT (E,) as described in Section 3.1 for the sEHR modality, and pro-trained GatorTron (Yang et al.) 2022) (E,) for the text modality. Global average pooling applied at the final layer for all three encoders obtain the representations. We use multi-layer perceptron (MTPP) for the projection heads.
4

---

Published in Transactions on Machine Learning Research (11/2023)
B,(+)
Po-rel()
Vayer
Embedking space
SEHn-BERT (GEHR ancoder)
MLP
Contrastive Loss
IetHR Proj
=EHR(a.)
E.(-)
F/
P Po-ee(
Va,es
Pro-ses
Ve,act
CNN (ECG encoder)
MLP (ECG Piej)
MLP Place
BCG()a)
B,C)
U/
Pe-nect)) (+)
Pt,act
GatarTron (Text encoder)
MLP
Contrastive
(Teat Proj)
Text(=))
aEHR-ECG-Text Embedding space (1
Figure 1: Overview of the BEHR-ECG-TE Text model pre-training The model takes input three modalities M {.m), m E M, i.e., ECG (xz), sEHR (zx), and text (xt). (- and denote the modality- specific encoder and modality-specifie representation respectively. Qt denotes the shared embedding space, where 2 represents the list of modalities that we embed into this space. For instance, & = es to denote joint ECG-sEHR space Dest (.)+e(-) denotes the projection head used to embed modality specific representation Um into sharod space Q2g Um.z denotes the reprosentation of the input modality 2m the shared space Ω2 The model is trained by applying contrastive loss between ECG and sEHR in the fine-grained ECG-sEHR, space (Res) and between ECG and text in the coarse- sarseegrained sEHR-ECCO EER--ECG-eettt space (Rget).
3.2.4 Multi-Modal Contrastive Objective
As mentioned before, we use FAC framework inspired from MMV (Alayrac et al, 2020) where ECG and sEHR are compared in fine-grained joint ECG-sEHR space Nas, while ECG is compared with text in coarse- grained joint sEHR-ECG- ECG-Text space Qget- Given a minibatch containing instances {w") [ we denote ", = Pm--z( Fm (nrr.) as the representation of the modality in the shared space Ω2 for the i-th instance. Following Zhang al. (2022), we define the contrastive objective bidirectionally. For example, in the case of contrastive loss between ECG and sEHR domains, the loss is directed from ECG to sEHR and vice-versa. In the context of contrastive objective,
consider N pairs of ECG-sEHR (æe: (.) as positive, while the
remaining 2 - N pairs are treated as negative. The same approach is applied to contrastive loss between ECG and text domains. Let sim(æ, y) = x² y/ 2, y E Rd., fee be the contrastive loss between ECG and sEHR, Les be the contrastive loss between ECG and text, des and der be scalar weights [0, 1), and IR+ be the temperature parameter. The combination of des and det gives the overall loss, denoted by L (Equation 3), and we aim to minimize this loss.
||||b||||| denote the cosine similarity between two vectors
N
exp (sim(1) os ()) + (1 des) log exp (sim(u "f (.ee//)
Les N
des log
Σ exp (sim(v) ont ((ceal/)) (1)
ΣM exp (sim(u) w 3/T))

---

Published in Transactions on Machine Learning Research (11/2023)
N
exp (sim(v 1i (1.st)/T) + (1 det)
exp (sim(vi sets vi sTT
Le =
det log
N
Σx=1 exp (simlre wh sT))
Ek=1 exp (sim( (o uk se)
(2)
L= L + Let
Figure illustrates the pre-training of the sEHR-ECG-Ter, model. See Appendix B.1 for more details about implementation and training. We also present shared space versus FAC spaces ablation in Appendix Wo provide the architecture of the shared space sEHR-ECG- model Appendix B.2
3.3 ECG-sEHR: Joint ECG and sEHR Pre-training
In the ECG-sEHR model, we pair ECG signals with structured EHRs as we describe in more detail in Scctions 4.2.1 and 4.2.2 We apply contrastive objective between ECG and sEHR modalities in joint ECG-
sEHR embedding space (Ree), where minimize the contrastive loss given in Equalion | ECG-sEHR model architecture in Appendix B.2
We provide the
3.4 ECG-Text: Joint ECG and Text Pre-training
In the ECG-Text model, we pair ECG signals with clinical text data from unstructured EHRs as we describe in more detail in Section 4.2.3 ECG and lext embeddings are jointly learned by applying the rastivve objective between ECG and text modalities in joint ECG-Text embedding space (Sex). We provide the ECG- Text model architecture in Appendix B.2 Following the notation introduced in Section 3.2
we minimize
the contrastive loss given by,
N
exp (sim(ve, vi())/T) + (1 Act) log Ek=1 exp (sim(ve ot o(T)
exp (sim(r), vi)e))
Let
Aet log
(4)
N
Σ exp (sim(v ot ((te)/T)
In this model, note that ECG and text are compared in joint ECG-Text space (Ret), which differs from the sEHR-ECG-Tex model where ECG and text are compared in the CGHR-ECG-Tex space (Aunt).
4 Experiments and Results
4.1 Dataset Splits Setup
We used EHR data of around 2.4 million patients from Mayo Clinic, USA consisting of around million ECGs to create datasets for pre-training and downstream tasks. These EHRs have undergone a rigorous do-identification process, guarantecing the utmost privacy and data security. These records have received approval from the Institutional Review Board (IRB) of Mayo Clinic, USA, ensuring compliance with ethical guidelines and regulations. Consequently, there are privacy data security issues associated with the use of these de-identified EHRs. We initially split all the patients into the global train, validation, and test sets in 60%, 5%, and 35% ratio, which are then used to create pro-training datascts and discase cohorts for downstream classification lasks. In particular train, validation, and test sels for pre-training and classification tasks are created by drawing the EHRs from the global train, validation, and test patients respectively. This approach ensures that we can effectively evaluate the quality of representations on downstream tasks, as the data of validation and test patients is not seen during the pre- pre-training phase. Consequently, all datasets across tasks have train, validation, and test split percentages roughly close to 60%, 5%, and 35% respectively.

---

Published in Transactions on Machine Learning Research (11/2023)
4.2 Pre-training Datasets
In this section, we describe the creation of following datasets for pre-training the proposed models: (i) sEHR sequences for pre-training the SEHR-BERT model, (ii) ECG-SEHR (2)) pairs for pre-training the ECG- sEHR model, and (iii) ECG-Text (z); pairs for pre-training the ECG-Text model. For sEHR-ECG-Tens model pre-training, paired with both sEHR and text data. See Appendix A.1 for details on the number of instances and patients used in different pre-training models.
construct triplets Ze, by considering (Z)) and (z,, 21) pairs that have ECG
4.2.1 Dataset for SEHR-BERT Pre-training
We use ICD-9 (International Classification of Diseases, Ninth Revision), 1CD-10 (International Classifica- tion of Discascs, Tenth Revision), CPT (Current Procodural Terminology), HCPS (Healthcare Common Procedures Coding System) codes, and medication prescriptions lo create the dataset for sEHR-BERT pre- training. Since ICD-9 codes differ from TCD-10 codes, but their corresponding text descriptions are similar, we map ICD-9 ICD-10 to maintain consistent phenotypic information. ICD-10 diagnosis codes are short- ened to the first three characters as keeping four more characters provides little to no extra information for large-scale pre-training. For example, the corresponding text. descriptions of TCD-10 diagnosis codes T26.0 and 126.9 are pulmonary emboliss with acute cor pulmonale and pulmonary embolism without acute cor pulmonale respectively, but these come under a common disease category, i.e., pulmonary embolism (126). Shortened ICD-10 diagnosis codes, ICD-10 procedure codes, CPT codes, HCPS codes, and medication pre- scriptions associated with at least 50 patients are included in the vocabulary, resulting a size of 28593. We present short. ICD-10 vs. full ICD-10 diagnosis codes ablation while keeping codes from other sources consistent in Appendix B To create the sEHR sequence for sEHR-BERT model pre-training randomly select one sequence of up to 512 consecutive medical codes from given patient's timeline. On average, the sequence length of the resulting datasct is 168.
4.2.2 ECG-sEHR Pairs Generation
To create the ECG-sEHR (I,) pairs, we first select an ECG of a given patient, Te, and consider all the shortened ICD-10 diagnosis codes, ICD-10 procedure codes, CPT codes, HCPS codes, and medication prescriptions associated with that patient within a period of one year prior, and one year subsequent, to the acquisition timestamp of that FCG. The medical codes restricted to this time range are arranged sequentially form the sEHR input sequence x,, The average length of the constructed sequences is 121.
4.2.3 Tex- Pairs Generation
F.Gss are paired with unstructured EHR text data derived from various sources, including ECG re- ports, ECHO reports, pathology reports, radiology reports, microbiology reports, clinical notes, and surgical notes, collectively referred to as "patient notes" in this work. Despite the GatorTron-bast (Yang et al. 2022) model's capacity to handle se- quences of up to 512 tokens, we were limited to work with maximun of 400 tokens due computing resource constraints. Given the abundance of text data within patient notes, implemented filter- ing process to extract only the most relevant infor- mation. Specifically, we selected patient noles that contained entities from a predefined list of biomed- ical entity types, such as discases, symptoms, procedures, medications, biomarkers, and gene mutations, using an in-house NLP model. The next step involves pairing ECGs with the selected patient notes. We employed two distinct methods for this purpose: report concatenation and entity concatenation.
-Bh-1
-50
Telt Ilepor
(brily
The figure shows the linkage of sEHR and text modalities with ECG modality within specific time windows: one year for the sEHR modality, one month for report concatenation, and one year for en- tity concatenation in the case of the text modality.
Figure 2:
7

---

Published in Transactions on Machine Learning Research (11/2023)
Report concatenation: For each patient's ECG, concatenate all selected patient notes that are available within one month around the ECG acquisition timestamp. This approach resulted in an average sequence length of 354 tokens after tokenization.
Entity concatenation: It's important to note that aligning ECGs with patient information spanning a more extended timeframe is advantageous for understanding ECG patterns more effectively. To effectively capture a broader spectrum of medical insights while adhering to token constraints, we concatenate only the identified entities from the patient notes within one-year timeframe around the ECG acquisition timestamp. This resulted in average sequence length of 266 after tokenization. Figure 2 illustrates how sEHR and text modalitios are linked with ECG modality within specific time windows around the ECG timestamp.
We use entity concatenation for the main results as it yielded better results when compared to report concatenation. We also present the report concatenation vs. entity concatenation ablation in Appendix E
4.3
Classification Datasets
In this section, we provide the details of the classification datasets that are used in linear classification and fino-tuning tasks. We evaluate the pre-trained models on both internal and external datasets.
4.3.1 Internal Datasets
We target six cardiac diseases whose diagnostic criteria using ECGs haven't been established in clinical practice, i.c., cither the patterns to identify these discases from ECG are not known or ECG is not the gold standard for definitive diagnosis. These include coronary atherosclerosis myocarditis, cardiac amyloidosis, pulmonary hypertension (PH), low left ventricular ejection fraction (low LVEF, ie., LVEF and atrial fibrilation in normal sinus rhythm (AFib in NSR). Those discases are diagnosed by other means and the diagnostic information available in EHRs. For example, diagnose PH, an invasive, right heart catheter- ization (RHC) procedure is performed and to identify low IVEF, an echocardiogram test. is performed. we utilize EHRs to associate ECGs with these diseases and generate labels. See Appendix A.2 for the summary of the discase labeling process, the number of ECGs, and the number of patients used.
4.3.2 External Datasets
We also evaluate all pre-trained models on two publicly available datasets: (i) PhysioNet2020 (Alday al,, 2020), which consists of collection of six 12-lead ECG datasets with varying signal lengths and sampling rales, (ii) Chapman (Zheng et al. 2020), which contains 10-second long 12-lead ECGs (see Appendix A.3 for more details). The discases in these datasots are commonly diagnosed by physicians directly using F.Gss unlike the diseases in our proprietary internal classification datasets. To replicate the state-of-the-art results on the PhysioNet2020 dataset presented by 3KG (Gopal et al. 2021), we followed their detailed procedure: (i) merge some of the conditions from the list of conditions due to their similarity, i.e., complete right bundle branch block and right bundle branch block, premature atrial contraction and supraventricular premature beats, premature ventrieular contractions and ventricular premature beats, 1st-degree atrioventricular block and prolonged PR interval, and evaluate on 23 distinct classes; (ii) resample the ECG signals to 500Hz; (iii) take non-overlapping 5-second long crops from each ECG record exhaustively and associate those with the label of the original record; (iv) split the dataset into 80%, 10,, and 10% for training, validation, and testing respectively; (v) train 23-class multilabel classification model. For the Chapman dataset, in line with Kiyasseh et al. (2021), we merge 11 cardiac arrbythmia conditions of the dataset into major classes and split: the datasct into 60%, 20%, and 20% for training, validation, and testing respectively.
4.4 Baseline Models
Random initialization models. We train binary classification models on all individual discases from scratch using the same ECG encoder that was used during pre-training, by randomly initializing the weights.
ECG-only contrastive learning models. We compare our models with the current state-of the-art ECG-only self-supervised learning models. In particular, we compare against the 3KG (Gopal al. 2021
8

---

Published in Transactions on Machine Learning Research (11/2023)
CLOCS(CMSC) (Kiyassch et al. 2021 and PCLR (Diamant al. 2022 models. For identical comparison, we pre-train all three models using the same global splits that we used for the multi-modal contrastivv pre-training but with only ECG signal as input and we also use the same ECG encoder that we used while pre-training multi-modal contrastive learning models.
4.5 Downstream Tasks and Results
In this section, we evaluate the pre-trained models' transfer learning capabilities and compare them with different basclinc methods on various downstream tasks such as classification, zero-shot retricval, and out- of-distribution (OOD) detection.
4.5.1 Classification Tasks
We evaluate the pre-trained models on linear classification and fine-tuning tasks. We evaluate the represen- tation quality by extracting embeddings from the pre-trained ECG encoder by passing FCG waveform input and training logistic regression models various cardiovascular discasos using both internally created and publicly available datasets as outlined in Section 4.3 In fine-tuning lasks, we add classification head (MLP) on top of the pre-trained ECG encoder and fine-tune the entire network. We compare our pre-trained models with various baseline models as described in Section 4.4 One of the most useful applications of pre- trained models is in providing downstream tasks with data efficiency, which ensures consistent performance with reduced training data. This is very valuable when a large amount of labeled data is not available due to the low prevalence of the diseases or is too expensive to procure. To demonstrate this, we create different fractions (1%, 10%, 100%) of the training set by maintaining the original discase prevalence. For low-data diseases such as coronary atherosclerosis, myocarditis, and cardiac amyloidosis, we drop 1% split due to small dataset sizes. We use AUROC and AUPRC as our evaluation metrics. We conduct each experiment using five random seeds and report the mean and standard deviation. For the 1% and 10% splits, we conducted experiments with five distinct fractional splits derived from the original training split (100%).
Results and Discussion. Table 1 shows the linear classification results on external datasets. Table 2 shows classification results on coronary atherosclerosis, myocarditis, and cardiac amyloidosis diseases (low- data) and Table 3 shows classification results on pulmonary hypertension, low LVEF, and AFib in NSR diseases (high-data). We observed the following findings: (i) Linear classifiers trained using representations obtained from our pre-trained models consistently outperform all baseline models by large margins across different fractions for all diseases. (ii) Fine-tuned classification models initialized with our pre-trained models' weights outperform all baseline models by large margin in low-data environments and small margin in high-data environments. (iii) Classification models trained with only 10% of the training data using our pre- trained models produce results that are as good as or better than those obtained by training on the entire dataset with baseline models. This demonstrates the effectiveness of our pre-trained models in achieving data efficiency. (iv) When utilizing the complete training dataset, on the linear classification task, our ECG- model achieves an AUROC score of 0.915 on PhysioNet2020, surpassing the SOTA performance reported in Gopal al. 2021 (3KG) by 2.5% (0.915 vs. 0.890). Additionally, on the Chapman dataset, it achieves an AUROC score of 0.990, surpassing the SOTA performance reported in Kiyasseh el. al. (2021) (CLOCS) by 3.1% (0.990 vs. 0.959). It's worth noting that, equal grounds, we surpass 3KG by 4.3% (0.915 vs. 0.872) and CLOCS by 4.4% (0.990 vs. 0.946). (v) We hypothesize that the main reason for the poor performance of ECG-only contrastive learning models on internal datasets is their exclusive dependence on ECGs for learning representations, i.e., comparing different instances of ECG data. This may not be sufficient to learn the complex patterns of various medical conditions. In contrast, our methods align ECGs with EHR data, providing rich contextual information about a patient's health history, including diagnoses, procedures, medications, and more. This approach is beneficial for learning ECG patterns more effectively.
Comparison between ECG-SEHR and ECG-Text models. The ECG-sEHR model demonstrates superior performance on internal datascts, whereas the ECG-Text model excels on external datasets. We hypothesize that this difference arises from the fact that external datasets, such as PhysioNet2020 and Chap- man, primarily comprise diseases that are commonly diagnosed from ECGs (i.e., arrhythmias, conduction
9

---

Published in Transactions on Machine Learning Research (11/2023)
Table 1: Linear classification results (AUROC, mean and standard deviation over runs with different ran- dom seeds) for PhysioNet2020 (23 classes) and Chapman (4 classes) datasets. Results within 95% confidence intervals of the best result are shown in bold.
PhysioNet2020
Chapman
Method
1%
10%
100%
1%
10%
100%
Superoised baseline
Random Tnit.
0.684 0.024
0.842 +: 0.006
0.913 + 0.006
0.532 + 0.031
0.964
0.003
0.987 +: 0.002
ECG-only SSL
3KG
0.774 0.001
0.837 0.003
0.872 + 0.000
0.934 + 0.008
0.968
0.001
0.984
0.000
CLOCS(CMSC)
0.770 + 0.005
0.836 = 0.002
0.864 + 0.000
0.876 + 0.019
0.936
+: 0.001
0,946 = 0.000
PCLR
0.721 0.003
0.798 0.003
0.834 + 0.001
0.722 0.007
0.821
0.004
0.872 0.000
Our models
sEUIR-ECG-Text
0,813 +: 0.005
0.882 = 0.003
0.911 +: 0.000
0.957 +: 0.004
0.979 +: 0.001
0.985 +: 0.000
ECG-sEHR
0.788 0.005
0.806 = 0.002
0.896 + 0.000
0.937 0.010
0.909
0.000
0.976 0.000
ECG-Text
0.820 0.003
0.887 0.001
0.915 1 0.000
0.974 0.002
0.987 0.001
0.990 0.000
Table 2: Results (AUROC, mean and standard deviation over 5 runs with different random seeds) for coronary atherosclerosis, myocarditis, and cardiac amyloidosis classification tasks: (a) linear classification, (b) fine-tuned classification. Results within 95% confidence intervals of the best result are shown in bold. See Appendix D for AUPRC metrics.
(a) Linear classification
Coronary atherosclerosis
Myocarditis
Cardinc amyloidosis
Method
10%
100%
10%
100%
10%
100%
Supervised baseline
Random Iuit.
0.772
0.012
0.831
0.004
0.775
0.019
0.868
0.003
+ 0.004
0.945
0.001
ECG-only SSL
3KG
0.722 = 0.008
0.786
0.000
0.699 + 0.022
0.808 0.000
0.809 + 0.009
0.906
0.000
CLOCS(CMSC)
0.743 + 0.004
0.801
0.000
0.662 +: 0.021
0.783 0.000
0.876 +: 0.006
0.918
0.000
PCT.R
0.761 0.004
0.825
0.000
0.022
0.818
0.000
0.006
0.000
Our models
SEHR-ECG-Text
0.840 = 0.003
0.890 + 0,000
0.860 +\2 0.019
0.901
0.001
0.934 1: 0.007
0.960
0.000
ECG-sEHR
0.836 0.011
0.891
0.000
0.859 0.011
0.896 0.000
0.932 + 0.006
0.959 0.000
ECG-Text
0.821 0.006
0.875 0.000
0.785 0.010
0.881 0.000
+ 0.006
0.949 0.000
(b) Fine-tuned classification
Coronary atheroaclerosis
Myocarditis
Cardiac amyloidosis
Method
10%
100%
10%
100%
10%
100%
Supervised baseline
Random Init.
0.772 + 0.012
0.831 + 0.004
0.775 + 0.019
0.868
0.003
0.912 + 0.004
0.945 0.001
ECG-only SSL
3KG
0.751
0.007
0.823 0.007
0.725 + 0.008
0.853 0.015
0.900 + 0.007
0.945 0.002
CLOCS(CMSC)
0.782
0.005
0.815
0.009
0.716
0.024
0.857
0.013
0.912 + 0.006
0.948 0.001
PCLR
0.812 1: 0.004
0.828 1. 0.005
0.722 1: 0.032
0.853 0.018
0.924 1: 0.003
0.951 0.001
Our models
sEHR-EOG-Text
0.880
0.003
0.888 0.005
0.862 + 0.005
0.905 0.003
0.948 + 0.003
0.961 0.002
ECG-sEHR
0.881
0.003
0.892 0.001
0.866 + 0.008
0.902 0.003
0.949 + 0.001
0.961 0.002
ECG-Text
0.871 = 0.002
0.880 + 0.001
0.857 + 0.008
0.892 0.004
0.940 4: 0.002
0.959 0.001
10

---

Published in Transactions on Machine Learning Research (11/2023)
Table 3: Results (AUROC, mean and standard deviation over 5 runs with different random seeds) for pulmonary hypertension, low LVEF, and AFib in NSR classification tasks: (a) linear classification, (b) fine- tuned classification. Results within 95% confidence intervals of the best result are shown in bold. See Appendix D for AUPRC metrics.
(a) Linear classification
Puluunary Ly Appertansio
LNEP
Alil NSR
Mothoo
1%
10%
100%
156
30%
1002
156
10%
100%
Superwised baseline
Romlom Init.
0,805
0.008
0.857
0.006
0,927 0,002
0,857 0.006
0,919 0.012
0.944 + 0,000
0.844 0.613
0.896 # 0.002
0,922
0.001
ECG-only SSL
3KG
0.779
0,018
0,802
0.002
0,874 0,000
0.846 0.016
0,902
0.914 0.000
0.824
00009
0.886
0.000
0.870
0.000
(IDCS(CMSC)
0,782
1: 0.021
0,857
1: 0.002
0.872 1: 0.000
0.861 =: 0.011
0.905
5: 6.002
0,016 1: 0.000
0.118
0.009
0.862 1: 0.000
0.868
0.000
PCLR
0.845
0.016
0,894
0.001
0,507
0.006
0.927
0.000
0,536
0.000
0.007 0.863
0.000
0.904
0.000
Our models
FFN-FFGGTett
0.900
0.005
0.9832
0.001
0.009 0.000
0.911
0.013
0.941
0.001
0.061
0.000
0.902
0.008
0.928
0.001
0.031
0.000
ECG sEII
0.899
0.007
0.803
0.040
0.000
0.910
0.001
0.951
0.000
0.007
0.930
0.001
0.933
0.000
FOG-TTect
0.889
0.000
0.921 + 0.001
0.931 + 0.000
0.501 0.014
0.038 0.001
0.947 0.000
0.891
0.008
0.923
0.000
0.925
0.000
(b) Fine-tuned classification
Puluumary ly aepertaniion
LNEP
Abil NSR
Method
15
10%
100%
156
30%
100%
156
10%
100%
Supereised baseline
Bandom lnit.
0,805
0,008
0,857
0,005
0,927 0,002
0.857
0.006
0,019 0.002
0.944 0,000
0.844
0.013
0.896 0.002
0,022 0.001
ECG-onll SSL
0.801
6.000
0.925
0.007
0.922
0.945
0.000
0.82
0.001
0.922
CLOCS(CMSC)
0.833
0.002
0,892 1 0.000
0,927 1. 0.001
0.800
0.002
0:020
0.002
0,945 1: 0.001
0.869
0.005
0.897 1: 0.002
0,020
0.001
PCLR
0.863
0.010
0.911 0.002
0.933 0.001
0.002 0.903
0.937
0.002
0.949
0.000
0.74 0.006
0.906
0.002
0,822
Our mundele
0.916
0.004
0.834 +: 0.001
0.038 :: 0.003
0.933
0.002
0.983
0.003
0.049
0.001
0.900
0.008
0.919 :: 0.002
0.926
0.001
ECG-EER
0.917
0.005
0.985
0.940 0.001
0.932
0.002
0.949
0.005
0.920
0.001
FOG-Test
0.907 0.000
0.920 + 0.001
0.931 + 0.001
0.927
0.004
0.938
0.001
0.948
0.001
0.001 + 0.008
0.916 + 0.003
0.926 0.002
blocks, etc.). These diagnoses are well-documented in the textual modality, particularly within ECG re- ports. As a result, the ECG-Text model performs best on these external datasets. In contrast, our internal datasets contain diseases for which labels are derived from EHR data. These labels are better captured by the sEHR modality. Therefore, the ECG-SEHR model outperforms the ECG-Text model when applicd to internal datasets. It's worth noting that the SEHR-ECG-TeR model demonstrates strong generalization across all internal datasets and achieves comparable performance with the ECG-Text model in the case of external datasets, i.c., PhysioNet2022 and Chapman, it is trained with both sEHR and text modalities. Furthermore, it offers the advantage of comparing different modalities for retrieval tasks.
4.5.2 Retrieval Tasks
Following Zhang al. 2022); we also evaluate the pre-trained models on two zero-shot retrieval tasks: (i) Zero-shot ECG-ECG Retrieval, and (ii) Zero-shot Tex Retrieval. We used data only from the global test split to create the queries and candidates for the retrieval tasks. For given query, we rank the candidates by computing the cosine similarity between the representations of the query and the candidates obtained from pre-trained encoders. For the Text-ECG retricval task, we obtain ECG and text embeddings from shared ECG- Text embedding space Qet- We report the precision@k metric for k=100, 500, and 1000, which represents the percentage of top k ranked candidates that are relevant the query.
Zero-shot ECG-ECG Retrieval. We take 1000 different ECGs as search queries for each of the 41 cardiovascular conditions that, are based on ECG reports. For every condition, we select 100,000 candidate ECGs, of which 10,000 are classified as positive for the condition and 90,000 are classified as negative for the condition. The query ECGs and positive candidate ECGs are completely exclusive.
11

---

Published in Transactions on Machine Learning Research (11/2023)
Zero-shot Text-ECG Retrieval. We take 1000 dis- tinct ECG reports search queries for each of the 41 cardiovascular conditions. For each of the conditions, we take 100,000 ECGs, out of which 10,000 ECGs show the condition and 90,000 FCGs have no connectiion to the condition. We also make sure that no ECG cor- responding to the 1000 distinct ECG report queries is chosen candidate for the positive set.
Table 4: Zero-shot ECG-ECG retrieval and Text- ECG retrieval results. The random category re- sults are from random guesses. precision@kk
P@k denotes
ECG-ECG Resriemi
Tex-EEG Retrieval
Method
10
P7500
P811000
P6110
P0000
Random
10.00
10.00
10.00
10.00
10.00
10.00
Table ! shows the zero-shot ECG-ECG retrieval and ECG-Text retrieval results. Our ECG-Text model out- performs the random guess method and ECG-only con- trastive learning models by large margin on both tasks.
PCLR
38.41
35.34
33.74
3KG
40.35
37.34
35.74
CLOCS
41.13
37.63
35.82
RCG-Taxt
55.18
49.47
73.02
66.92
63.58
4.5.3 Duttof-Distribution Detection
Tt is observed that the representations learned via self- supervised learning techniques help to better distinguish be- tween in-distribution (IND) and out-of-distribution (OOD) datasets. We demonstrate this using representations obtained from our ECG-sEHR model to differentiate between two dis- parate ECG datasets. We take the proprietary ECG pul- monary hypertension (PH) cohort as the IND dataset and Holter ECGs (ECG recorded continuously over 24 hours longer) from the open-source St Petersburg INCART 12-lead Arrhythmia Database (Tihonenko et al. 2008) as the OOD dataset. Non- Non-overlapping 10-second long segments are taken from Holter ECGs and resampled to 500Hz to be consistent with ECGs from the IND PH dataset. We use the relative Mahalanobis distance (RMD) (Ren et al. 2021 method which is based on the Mahalanobis distance of embeddings from the distribution of the nearest pre- dicted class, to determine whether the data is in-distribution or out-of-distribution. We compare the results obtained using representations extracted from the ECG-encoder of the pre-trained ECG-sEHR. model with that of the representations obtained from the penultimate layer of the PH binary classifier trained from scratch on the PH cohort and show that the rejection rate at different significance levels is much higher in the case of ECG-sEHR model. The results are given in Table 5 which clearly shows that generic ECG representations are better at detecting out-of-distributiio data compared to disease-specific representations.
Table 5: Out-of-distribution detection re- sults using ECG representations obtained from the PH disease model and generic ECG representations obtained from the ECG- sEHR model. sig. denotes significance level.
Metric
ECG-sEHR
PH
Rejection at 1% sig. (%)
13.94
10.35
Rejection at 5% sig. (%)
19.67
29.87
IND vs. OOD AUROC
0.757
0.620
5 Conclusion
Our work introduces series of three multi-modal contrastive learning models. These models leverage both structured and unstructured EHRs to produce high-quality ECG representations. We have detenst.rate that our pre-trained models outperform randomly initialized models and other ECG-only contrastive learning models by a wide margin on classification and retrioval tasks. Specifically, we perform the classification tasks using ECGs on cardiovascular diseases whose definitive diagnoses are obtained from more expensive and /r invasive tests in clinical settings. This is significant breakthrough as ECG tests are widely available, non- invasive, and less expensive. Furthermore, our ECG representations have been shown excel in detecting out-of-distribution data when compared to discaso-specific representations.
6
Future Work
In this work, we make use of both structured EHRs and unstructured EHR text data to learn ECG rep- resentations. However, there are additional modalities present in unstructured EHRs such as images (MRI scans, CT scans, X-rays, and histopathology images related to cardiology), videos (echocardiograms//i) heartt ultrasounds), and time-series signals (heart and lung sounds), which can provide even more meaningful infor-
12

---

Published in Transactions on Machine Learning Research (11/2023)
mation through multi-modal contrastive learning. Another interesting future work would be the integration of federated learning frameworks to leverage multi-instiiatiinnal medical data. This approach aims to capture a more diverse range of patient information, leading to enhanced ECG representation learning. While the disease models presented in this study undergo training and testing using real-world datascts, it is of ut- most importance to conduct clinical validation across a diverse set of health systems before deploying them. This ensures that the models are equitable, unbiased, and trustworthy. We hope our work will serve as an inspiration for future endeavors in harnessing multi-modal EHR data to learn robust ECG representations.
7 Acknowledgements
We would like to thank Dr. Rickey E. Carter for insightful discussion and valuable feedback on this study.
References
Demilade Adedinsewo, Rickey E Carter, Zachi Attia, Patrick Johnson, Anthony H Kashou, Jennifer L Dugan, Michael Albus, Johnathan M Sheele, Fernanda Bellolio, Paul A Friedman, el al. Artificial intelligence- enabled eeg algorithm to identify patients with left ventricular systolic dysfunction presenting the emergency department with dyspnea. Circulation: Arrlythmia and Electrophyysooogyy 13(8):e008437, 2020.
Joseph C Ahn, Zachi I Attia, Puru Rattan, Aidan F Mullan, Seth Buryska, Alina M Allen, Patrick S Kamath, Paul A Friedman, Vijay H Shah, Peter A Noseworthy, et al. Development of the ai-cirrhosis-eng score: an deep learning model in cirrhosis. The American Journal of Gastroenterology, 117 (3):424-432, 2022.
electrocardiogrambbre
Jean Jean-Baptiste Alayrac, Adrià Recasens, Rosalia Schneider, Relja Arandjelovié, Jason Ramapuram, Jeffrey De Fauw, Lucas Smaira, Sander Dieleman, and Andrew Zisserman. Self-Supervised MultiModal Versatile Networks. In Newr/PS, 2020.
Erick A Perez Alday, Annie Gu, Amit J Shah, Chad Robichaux, An-Kwok Ian Wong, Chengyu Liu, Feifei Täu, Ali Bahrami Rad, Andoni Flola, Salman Seyedi, et al. Classification of 12-lead eegs: the phys- ionet /com///tiiiing in cardiology challenge 2020. Physiological measurement, 41(12):124003, 2020.
Zachi Attia, Suraj Kapa, Francisco Lopez-Jimencz, Paul M McKie, Dorothy Ladewig, Gaurav Satam, Patricia A Pellikka, Maurice Enriquez-Sarano, Peter A Noseworthy, Thomas M Munger, et al. Screening for cardiac contractile dysfunction using an artificial intelligence-nna enabled electrocardiogram. Nature medicine, 25(1):70 74, 2019a.
Zachi Attia, Suraj Kapa, Xiaoxi Yao, Francisco Lopez-Jimenes Tarun Mohan, Patricia A Pellikka, Rickey F. Carter, Nilay D) Shah, Paul A Friedman, and Peter A Noseworthy. Prospective validation of a deep learning electrocardiogram algorithm for the detection of left ventricular systolic dysfunction. Journal of cardiovascular electrophyssology, 30(5):668 674, 2019b.
Zachi I Attia, Peter A Noseworthy, Francisco Lopez-Jimenez, Samuel J Asirvatham, Abhishek J Deshmukh, Bernard J Gersh, Rickey E Carter, Xiaoxi Yao, Alejandro A Rabinstein, Brad J Erickson, et al. An artificial intelligence-enabled eeg algorithm for the identification of patients with atrial fibrillation during sinus rhythm: a retrospective ive analysis of outcome prediction. The Lancel, 394(10201) (::1000)):8611100 8666-1867 2019c.
Shekoofeh Azizi, Basil Mustafa, Fiona Ryan, Zachary Beaver, Jan Freyberg, Jonathan Deaton, Aaron Loh, Alan Karthikosalingam, Simon Kornblith, Ting Chen, ct al. Big self-supervised models advance medical image classification. In Proceedings of the IEEE/CVF International Conference on Compuler Vision, pp. 3478 3488, 2021.
Shekoofch Azizi, Laura Culp, Jan Freyberg, Basil Mustafa, Sebastien Baur, Simon Kornblith, Ting Chen, Patricia MacWilliams, Sara Mahdavi, Ellery Wulczyn, al. Robust and efficient medical imaging with self-supervision. arXio preprint ar Xiw:2205.007223 2022.
13

---

Published in Transactions on Machine Learning Research (11/2023)
Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli. wav2vee 2.0: A framework for self-supervised learning of speech representations. Advances in neural information processing systems, 33: 12449 12460, 2020.
Shruthi Bannur, Stephanie Hyland, Flora Liu, Fernando Pérez-García, Maximilian Tsse, Daniel Coelho de Castro, Benedikt Boecking, Harshita Sharma, Kenza Bouzid, Anja Thieme, Anton Schwaighofer, Maria Teodora Wetscherek, Matthew Lungren, Aditya Nori, Javier Alvarez- Valle, and Ozan Oktay. Learn- ing to exploit temporal structure for biomedical vision-language processing. Tn The TEEE/CVE Conference on Computer Vision and Pattern Recognition (CVPR), June 2023.
Benedikt. Boecking, Naoto Usuyama, Shruthi Banmur, Daniel Coelho de Castro, Anton Schwaighofer, Stephanie Hyland, Maria Teodora Wetscherek, Tristan Naumann, Aditya Nori, Javier Alvarez- Valle, Hoi- fung Poon, and Ozan Oktay. Making the most of text semantics to improve biomedical vision-language processing. In The European Conference on Computer Vision (ECCV), October 2022.
J Martijn Bos, Zachi I Attia, David E Albert, Peter A Noseworthy, Paul A Friedman, and Michael J Ackerman. Use of artificial intelligence and deep neural networks in evaluation of patients with electrocar- diographically conccaled long qt syndrome from the surface 12-lcad eloctrocardiogram. JAMA cardiology. 6(5):532 538, 2021.
Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsu- pervised learning of visual features by contrasting cluster assignments. Advances in neural in information processing systems, 33:9912 9924, 2020.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pp. 1597 1607. PMLR, 2020a.
Ting Chen, Simon Kornblith, Kevin Swersky, Mohammad Norouzi, and Geoffrey E Hinton. Big self- supervised models are strong semi-supervised learners. Advances in neural information processing systems, 33:22243 11222243-22255, 2020b.
Joseph Y Cheng, Hanlin Goh, Kaan Dogrusor, Oncel Tuzel, and Erdrin Azemi. Subject-aware rastivvv learning for biosignals. arXiv preprint or arXim:2007.04877 2020.
Georgios Christopoulos, Jonathan Graff-Radford, Camden I, Lopez, Xiaoxi Yao, Zachi T Attia, Alejandro A Rabinstein, Ronald C Petersen, David S Knopman, Michelle M Mielke, Walter Kremers, et al. Artificial intelligence electrocardiogrralyy to predict incident atrial fibrillation: A population-based study. Circula- tion: Arrhythmia and Electrophysiology. 13(1)):e0,9355, 2020.
Ozan Ciga, Tony Xu, and Anne Louise Martel. Self supervised contrastive learning for digital histopathology. Machine Learning with Applications, 7:100198, 2022.
Michal Cohen-Shelly, Zachi Attia, Paul A Friedman, Saki Ito, Benjamin A Essayagh, Wei- Yin Ko, Dennis H Murphree, Hector T Michelena, Maurice Fnriquez-Sarano, Rickey E. Carter, et al. Flectrocardiogram screening for aortic valve stenosis using artificial intelligence. Europeon heart journal, 42(30))2885-2888) 2021.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint ar arXiv:1810.04800 04800,, 2018.
Nathaniel Diamant, Erik Reinertsen, Steven Song, Aaron Aguirre, Collin M Stultz, and Puneet Batra. Patient contrastive learning: A performant, expressive, and practical approach to electrocardiogram mod- eling, PLoS Computational Biology, 1802)ee000962,, 2022.
Conner D Galloway, Alexander V Valys, Jacqueline B Shreibati, Daniel L Treiman, Frank L Petterson, Vivek P Gundotra, David E Albert, Zachi I Attia, Rickey Carter, Samuel J Asirvatham, et al. Devel- opment and validation of a deep-learning model to screen for hyperkalemia from the electrocardiogram. JAMA cardiology, 4(5):428 436, 2019.
14

---

Published in Transactions on Machine Learning Research (11/2023)
Shashank Goel, Hritik Bansal, Sumit Bhatia, Ryan A. Rossi, Vishwa Vinay, and Aditya Grover. CyCLIP: Cyclic contrastive language-image pretraining. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Processing Systems, 2022.
Bryan Gopal, Ryan Han, Gautham Raghupathi, Andrew Ng, Geoff Tison, and Pranav Rajpurkar. 3kg: con- trastive learning 12-lead electrocardiograms using waapooooiccalll-nppiieee augmentations. Machine Learning for Health, pp. 156- 167.167..08 PMLR, 2021.
Priya Goyal, Mathilde Caron, Benjamin Lefaudeux, Min Xu, Pengchao Wang, Vivek Pai, Mannat Singh, Vitaliy Liptchinsky, Ishan Misra, Armand Joulin, et al. Self-supervised pretraining of visual features the wild. arXiv preprinl Xrv:211033. 01 018, 2021.
Martha Grogan, Francisco Lopez-Jimenez, Michal Cohen-Shelly, Angela Dispenzieri, Zachi T Attia, Omar F Abou Ezzedine, Grace Lin, Suraj Kapa, Daniel D Borgeson, Paul A Friedman, et al. Artificial intelligence- enhanced electrocardiogram for the early detection of cardiac anyloidosis. In Mayo Clivic Proceedings, volume 96, pp. 2768 2778. Elsevier, 2021.
Awni Y Hannun, Pranav Rajpurkar, Masoumeh Haghpanahi, Geoffrey H Tison, Codie Bourn, Mintu P Turakhia, and Andrew Y Ng. Cardiologist-Iove arrhythmia detection and classification in ambulatory electrocardiograms using a deep neural network. Nature medicine, 25(1):65 69, 2019.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Kaiming Hc, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervisod visual representation learning. In Proceedings of the IEEE/CVF conference on compuler vision and paltern recognition, pp. 9729-9738, 2020.
Shih-Cheng Huang, Liyue Shen, Matthew P Lungren, and Screna Yeung. Gloria: A multimodal global- local representation learning framework for label-efficient medical image recognition. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp.
2021.
Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig, Scaling up visual and vision-language representation learning with noisy text supervision. In International Conference on Machine Learning, pp. 4904-4916. PMLR, 2021.
Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. Supervised contrastive learning. Advances in neural information processing systems, 33:18661 18673, 2020.
Dani Kiyasseh, Tingting Zhu, and David A Clifton. Clocs: Contrastive learning of cardiac signals across space, time, and patients. In International Conference Machine Learning, pp. 1606-5615 PMLR, 2021.
Wei-Yin Ko, Konstantinos C Siontis, Zachi I Attia, Rickey E Carter, Suraj Kapa, Steve R Ommen, Steven J Demuth, Michael Ackerman, Bernard J Gersh, Adelaide M Arruda-Olson, et al. Detection of hypertrophic cardiomyopathy using convolutional neural network-enabled electrocardiogram. Journal of the American College of Cardiology, 75(7):722-77) -73,, 2020.
Junman Li, Pan Zhou, Caiming Xiong, and Steven Hoi. Prototypical contrastive learning of unsupervised representations. In International Conference on Learning Representations, 2021.
Yikuan Li, Shishir Rao, José Roberto Ayala Solares, Abdelaali Hassaine, Rema Ramakrishnan, Dexter Canoy, Yajie Zhu, Kazem Rahimi, and Gholamreza Salimi-Khorshide Behrt: transformer for electronic health records. Scientific reports, 10(1):7155, 2020.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations, 2019.
15

---

Published in Transactions on Machine Learning Research (11/2023)
Ming Y. Lu, Bowen Chen, Andrew Zhang, Drew F. K. Williamson, Richard J. Chen, Tong Ding, Long Phi Le, Yung-Sung Chuang, and Faisal Mahmood. Visual language pretrained multiple instance zero-shot transfer for histopathology images. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 19764-19775, June 2023.
Temesgen Mchari and Nils Strodthoff. Self-supervised representation learning from 12-lead ecg data. Com- pulers in biology and medicine, 141:105114, 2022.
Ishan Misra and Laurens van der Maaten. Self-supervieed learning of pretext-invariant representations. Tn Proceedings of the IEEE/CVF conference on computer vision and pottern recognition, pp. 6707 7717, 2020.
Jungwoo Oh, Hyunseung Chung, Joon-myoung Kwon, Dong-gyun Hong, and Edward Choi. Lead-agnostic sclf-supervised learning for local and global representations of clcctrocardiogram. In Conference on Health, Inference, and Learning, pp. 338 353. PMLR, 2022.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint arXi::1807.0033 03748, 2018.
Chao Pang, Xinzhuo Jiang, Krishna S Kalluri, Matthew Spotnitz, RuiJun Chen, Adler Perotte, and Karthik Natarajan. Cehr-bert: Incorporating temporal information from structured chr data improve prediction tasks. In Machine Learning for Health, pp. 239-260. PMLR, 2021.
Raphael Poulain, Mehak Gupla, and Rahmatollah Beheshti. Few-shot learning with semi-supervised trans- formers for electronic health records. In Zachary Lipton, Rajesh Ranganath, Mark Sendak, Michael Sjoding, and Serena Yeung (eds.), Proceedings of the 7th Machine Learning for Healtheare Conference, volume 182 of Proceedings of Machine Learning Research, pp. 173. PMLR, 05-06 Aug 2022.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference machine learning, pp. 8748 8763. PMLR, 2021.
Aniruddh Raghu, Payal Chandak, Ridwan Alam, John Guttag, and Collin Stultz. Contrastive pre-training for multimodal medical time series, In NewrIPS 2022 Workshop on Learning from Time Series for Health, 2022.
Laila Rasmy, Yang Xiang, Ziqian Xic, Cui Tao, and Degui Zhi. Med-bert: pretrained contextualized em- beddings on large-scale structured electronic health records for disease prediction. NPJ digilal medicine, 4(1):86, 2021.
Jie Ren, Stanislav Fort, Jeremiah Liu, Abhijit Guha Roy, Shreyas Padhy, and Balaji Lakshminarayanan. A simple fix mahalanobis distance for improving near-ood detection. arxiv preprint orXiv:2106.00m 2021.
Konstantinos C Siontis, Peter A Noseworthy, Zachi I Attia, and Paul A Friedman. Artificial intelligence- enhanced electrocardiograply in cardiovascular disease management. Nature Reviews Cardiology, 18(7): 465-478, 2021.
Hari Sowrirajan, Jingbo Yang, Andrew Y Ng, and Pranav Rajpurkar. Moco pretraining improves represen- (ation and transferability of chest x-ray models. In Medical Imaging with Deep Learning, pp. 728 744.
PMLR, 2021.
Chetan L Srinidhi and Anne L Martel. Improving self-supervised learning with hardness-aware dynamic curriculum learning: an application to digital pathology. In Proceedings of the IEEE/CVE International Conference on Computer Vision, pp. 562-571, 2021.
Vikto Tihonenko, A Khaustov, S Ivanov, A Rivin, and E Yakushenko. St petersburg incart 12-lead arrhyth- mia database. PhysioBank Bank Physio Toolkit and PhysioNet, 2008.
16

---

Published in Transactions on Machine Learning Research (11/2023)
Geoffrey H. Tison, Jeffrey Zhang, Francesca N Delling, and Rahul C Deo. Automated and interpretable patient ecg profiles for disease detection, tracking, and discovery. Circulation: Cardiovascular Quality and Outcomes, ee00)e000 2019.
Rachael A. Venn, Xin Wang, Sam Freesun Friedman, Nate Diamant, Shaan Khurshid, Paolo Di Achille, Lu-Chen Weng, Scung Hoan Choi, Christopher Reeder, James P. Pirruccello, Pulkit Singh, Emily S. Lau, Anthony Philippakis, Christopher D. Anderson, Patrick T. Ellinor, Jennifer E. Ho, Puneet, Batra, and Steven A. Lubitz. Deep learning of electrocardiograns enables scalable human disease profiling. medRriv, 2022. doi: 10.1101 2000112022222 20.2.77..1.22287777
22283757.
Xiyue Wang, Sen Yang, Jun Zhang, Minghui Wang, Jing Zhang, Wei Yang, Junzhou Huang, and Xiao Han. Transformer-based unsupervised contrastive learning for histopathological image classification. Medical Image Analysis, 2022.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier- ric Cistac, Tim Rault, Rémi Louf, Morgan Funtowiez, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pp. 38 45, Online, October 2020. Association for Computational Linguistics.
Xi Yang, Nima PourNejatian, Hoo Chang Shin, Kaleb E Smith, Christopher Parisien, Colin Compas, Cheryl Martin, Mona G Flores, Ying Zhang, Tanja Magoc, et al. Gatortron: A large clinical language model to unlock patient information from unstructured elect health records. arXiv preprinl urXiv:2203.035. 2022.
Xiacxi Yao, David R Rushlow, Jonathan W Inselman, Rozalina G McCoy, Thomas D Thacher, Emma M Behnken, Matthew E Bernard, Steven L, Rosas, Abdulla Akfaly, Artika Misra, et al. Artificial intelligence- enabled electrocardiograns for identification of patients with low ejection fraction: a pragmatic, random- ized clinical trial. Nature Medicine, 27(5):815- 819, 2021.
Yuhao Zhang, Hang Jiang, Yasuhide Miura, Christopher D Manning, and Curtis P Langlotz. Contrastive learning of medical visual representations from paired images and text. In Machine Learning for Healthcare Conference, pp. 2 25. PMLR, 2022.
Jianwei Zheng, Jianming Zhang, Sidy Danioko, Hai Yao, Hangyuan Guo, and Cyril Rakovski. A 12-lead clectrocardiogram database for arrhythmia rescarch covering more than 10,000 patients. Scientific data, 7(1):48, 2020.
17

---

Published in Transactions on Machine Learning Research (11/2023)
Appendix
A
Dataset Details
A.1 Pre-training Dataset Details
Table 6: The number of FCGs and the number of patients used during pre-training of the proposed models, i.e, SEHR-BERT, SELLR--CG-Teee ECG-sEHR, and ECG-Text.
Train
Validation
Test
Total
Model
#ECGs
PPaticuts
ECCSs
Pattients
#ECGs
# Patients
#ECGs
Paatients
Global Splits
5,479,435
1,463,009
450,775
121,932
3,210,110
853,477
9,140,320
2,438,418
SEHR-BERT
1,167,991
07,333
Text
4,526,686
1,177,903
371 371,367
98,013
ECG-sEHR
4,553,278
1,196,478
373,649
90, 90,608
BCG-Text
5,416,467
1,423,999
445,342
118,628
Distribution of Time Difference Between ECG and SEHR Modalities
Distr bution of Time Difference Between ECG and Text Modalities
5M4
Number of ECGs (Aillions)
Number (Millans)
-2
2M
200
1M
08
06
enfference (Weeks)
(a) Time difference distribution between ECG and sEHR modalities.
(b) Time difference distribution between ECG and text modalities.
Figure 3: The figures (a) and (b) show the distribution of time difference between ECG and other modalities: (a) the distribution of time difference between ECG and sEHR modalities, (b) the distribution of time difference between ECG and text modalities.
Table 6 outlines the number of ECGs and the number of patients used during pre-training of the proposed models. Figure 3a shows the time difference distribution between ECG and sEHR modalities, while Figure 3b displays the time difference distribution between ECG and text modalities. The Y-axis in Figure 3a represents the number of ECGs associated with at least one medical code for the sEHR modality one- week intervals and the Y-axis Figure 3b represents the number of ECGs associated with least one biomedical entity (mentioned in Section 4.2.3) for the text modality in one- week intervals. The distributions are plotted using one-year time window (i.e., approximately 52 weeks) around FCG acquisition timestamp.
18

---

Published in Transactions on Machine Learning Research (11/2023)
A.2 Internal Classification Dataset Details
Table 7: Summary of the disease labeling process for coronary atherosclerosis, myocarditis, cardiac amyloi- dosís, pulmonary hypertension, and low IVEF. The Case column details the identification of patients with the specific disease, the Control column explains how control patients are produced for the corresponding disease, and the Time window indicates the time frame for associating ECGs with diagnoses. (d1 d2) de- notes the time frame extending d1 days before and d2 days after the first diagnosis timestamp. All ECGs of the control paticnts are taken into the control cohort. The abbreviations CAC, TTE, mPAP, TRV, and LVEF represent coronary artery calcium, transthoracic echocardiogram, mean pulmonary arterial pressure, tricuspid regurgitation velocity, and left ventricular ejection fraction, respectively.
Disenae
Crss
Control
Time window
Coronary Manual expert physicias wingg EHR Manual curation expert plesiciams usnng EIR data mPAP 25 manlig or TEV 2 3.4 ma/s IVEF 5 40
(365, 363)
MyocarEtis
history myucarditis
[7, 7]
Cardac amploidoeis
Patients who have undergone TTE and history of amyloddasi
(180, 180)
Pulmoonrr lopertension
mPAP 20 mualig or TRV 2.8 m/s
(30, 30)
Low IVEP
IVEP 40
(14, 14)
Table 8: Number of ECGs, number of patients, and disease prevalence in coronary atherosclerosis, myocardi- tis, cardiac amyloidosis, pulmonary hypertension, low LVEF, and AFib in NSR classification datasets.
Train
Validation
Test
Discase
#ECCss
Patients
Prev.(%) (%)
#ECGs
@ Patients
Prev.(%)
ECCGs
Pattients
Prev.
Coronary atherosclerosis
19,281
10,589
38.78
1,604
870
39.66
11,290
6,088
39.13
Myocarditis
53,432
52,299
0.97
4,366
4,260
1.17
31,715
30,984
1.02
Cardiac amyloidosis
34,465
20,011
8.54
2,795
2,462
8.04
20,071
17,508
8.51
Pulmonary hypertension
200,777
73,908
14.71
16,132
6,091
14.10
115,602
42,893
14.34
Low IVEF
166,702
166,7U2
7.58
13,814
13,814
7.69
97,109
97,109
7.48
AFib in NSR
1,455.626
514,871
7.28
42,627
42,627
7.45
301,022
301 .022
7.32
In this section, summarize the disease labeling process and outline the number of ECGs, the number of patients, and the discase prevalence used during training, validation, and testing of internal disease classification models. Table/7shww the summary of the disease labeling process for coronary at atherosclerosis, myocarditis, cardiac amyloidosis, pulmonary hypertension, and low LVEF. In the case of AFib in NSR, for the case-cohort, we eucompass all NSR ECGs: those from 30 days before the first occurrence of AFib, during the occurrences, and extending 30 days after the last occurrence of AFib. The control cohort comprises all NSR ECGs obtained from patients with evidence of AFib. Table g shows the number of ECGs, the number of patients, and the prevalence for each discase.
A.3 External Classification Dataset Details
In this section, we provide an overview of the number ECGs and the number of pationts in PhysioNet2020 and Chapman datasets (see Table D) The PeysioNet2020 dataset consists of ECG signals from multiple sources with varying signal lengths and sampling rates.
19

---

Published in Transactions on Machine Learning Research (11/2023)
Table 9: Details the PhysioNet2020 and Chapman datasets, including the number of ECGs, number of patients, signal lengths, and sampling rates.
Dataset
#ECGs
#Patients
Signal length
Sampling rate
PhysioNet2020
CPSC2018
6,877
6,877
6-60 secs
500 Hz
CPSC extra
3,453
3,453
6-60 secs
500 Hz
St Petersburg INCART
74
32
30 mins
257 Hz
PTB
516
516
1000Hz
PTB-XL
21,837
21,837
10 sees
500 Hz
Georgia
10,344
10,344
10 sees
500 Hz
Chapman
Chapman
10,646
10,646
10 sees
500 Hz
B Model Architectures and Implementation Details
B.1 Training Details
We execute all pre-training and classification tasks using 2 Nvidia V100 (16G) GPUs, However, for the pretraining tasks involving the text domain, we utilize 2 Nvidia A100 (40G) GPUs. All the original ECGs consist of 12 leads and are 10 seconds long with sampling rate of 500Hz. During training, use random crop of 5 seconds in length, i.c., 2500 samples. We optimize all pre-training and classification models with AdamW optimizer (Loshchilov & Hutter 2019 with (81, set to (0.9, 0.999).
Pre-training Details We initially pre-train SEHR-BERT as described in Section 3.1 For multi-modal contrastive pre-training, we initialize the sEHR encoder with SEHR-BERT model weights and the text encoder with GatorTron-base (Yang et al., 2022) model weights. GatorTron- (Yang et al., 2022 is a 345M-parameter language model pre-trained on large amounts of de-identified clinical notes (80B words) from the University of Florida Health System, having a vocabulary of size 50176. We use BERT and Megatron- BERT implementation offered by the Huggingface transformers library (Wolf et al.) 2020) for sEHR and text encoders respectively. For the ECG encoder, we use ResNet-like architecture (He et al., 2016) customized to dimension which consists of around 1M paramcters. The ECG encoder is initialized randomly. In joint pre-training, we freeze the first 3 and 18 layers of sEHR and text encoders respectively, and fine-tune the remaining layers. Following Zhang et al. (2022), we set to 0.1. We assign equal weighting to both the directions of contrastive learning, i.c., from ECG to sEHR and sEHR to ECG, similarly for ECG and text, i.c, (ees, Xex) is set to (0.5, 0.5). We used batch size of 256 and initial learning rate of lc-4 for models. For F.CG-oll contrastive learning models, we used batch size of 512 and an initial learning rate of 1e-3. The learning rate is reduced by a factor of 2 if the validation loss stops decreasing continuously for 2 epochs and we carly stop the training based on validation loss with an carly stopping paticnce of 10 epochs.
Classification Details For classification tasks, we add a two-layered MLP head on top of the ECG en- coder. We also add dropout layers after each hidden layer with a dropout probability of 0.2 for regularisation. A batch size of 128 is used for all classification models. We used initial learning rate of 1e-3 for random initialization training for all diseases. For fine-tuning, atherosclerosis and myocarditis tasks, and 1c-4 for cardiac amyloidosis, pulmonary hypertension, low LVEF, and AFib in NSR lasks. The learning rate is reduced by a factor of 2 if the validation score slops increasing continuously for 2 epochs and we early stop the training based on validation loss with an early stopping patience of 10 epochs. During fino-tuning, we initialize the ECG encoder with the pro-trained ECG encoder weights and warm up the classification head (MLP) for 1024 steps by freezing the backbone network weights and then fine-tuning the entire network. During prediction, we take 6 consecutive 5-second long crops with
used initial learning rate of 1e-3 for coronary
20

---

Published in Transactions on Machine Learning Research (11/2023)
stride of second from the original 10-second long ECG. The median of the predictions of these 6 crops is taken as the final prediction for computing the AUROC score.
B.2 Multi-Modal Architectures
In this section, we present the precise architectures of various multi-modal contrastive learning models. Figure 4a and Figure 4b show the architectures of the ECG-sEHR model, described in Section 3.3 and the ECG- Text model described in Section 3.4 respectively. Figure Ae shows the architecture of sEHR-ECG-Tess
shared space model mentioned in Section 3.2
800
100mm)
Desccing cow PlL
Texhko
(a) ECG-sEHR model architecture
(b) ECG-Text model architecture
Pioel
Temto
(c) sEHR-ECG--EE Tex- shared space architecture
Figure 4: The figures (a) and (b) show the architectures of bi-modal contrastive learning methods. (a) illustrates the architecture for the ECG-sEHR model, while (b) presents the architecture for the ECG-Text model. Figure (c) shows the shared space architecture of the SEHR-ECG-Toxt model, where contrastive loss between ECG and sEHR and, between ECG and text, applied in SEHR-ECG-TEE Text shared space (M
B.3 ECG Encoder Architecture
We use ResNet-llke architecture ure (He et. al. 2016) customized to 1 dimension for time series ECG signals. This consisted eight ID convolution layers based on the basic block of ResNet. Details of each layer are given Table [0] All convolutional layers employ batch normalization and ReLU activation function and a stride of 2. We add fully connected layers with hidden sizes 128 and 64 on top of the backbone CNN architecture for classification tasks. We the same architecture for all pre-training methods.
21

---

Published in Transactions on Machine Learning Research (11/2023)
Table 10: ECG encoder architecture used for all experiments. 1C, OC, and K represent the number of input channels, the number of output channels, and kernel size, respectively.
Layer
Layer type
IC
OC
K
1
Conv
12
32
2
Conv
32
32
5
3
Conv
32
64
4
Conv
64
64
3
5
Conv
64
128
6
Conv
128
128
7
Conv
128
256
3
8
Conv
256
256
C
Statistical Testing
We assess the statistical significance of our generalized model, sEHR-ECG-Text, by comparing it to each baseline using a two-sided t-test. This evaluation conducted on linear classification tasks with 10% training splits. Fach experiment repeated 10 times, with 10 different fractional splits derived from the original training data. Our model significantly outperformed all the baselines with the p-value less than 1e-5. We show the average AUROC score over 10 runs along with 95% confidence intervals in Table 11
Table 11: The table shows the average AUROC, with 95% confidence intervals in parentheses on linear classification tasks, i.e, coronary atherosclerosis, myocarditis, cardiac amyloidosis, pulmonary hypertension, low LVEF, and AFib in NSR, using 10% training splits. The results were calculated by conducting cach experiment 10 times, using 10 different fractional splits obtained from the original training split. The sEHR- ECG-Text model significantly outperforms all the baselines with the p- value less than 1e-5.
Coronary acheroscleross
Cardiae anyyoidosis
Pulmonary hypertension
Myscarditis
Low LVEF
AFib in NSR
Method
Supervised baseline
Random
0.770 (0.780, 0.779)
0.778 (0.767, 0.789)
(0.907, 0.918)
0.886 (0.882, 0.890)
(0.915, 0,920)
(0.895, 0.898)
ECG-only SSL
3KG
0.721 (0.715, 0.727)
0.686 (0.669, 0.704)
0.869 (0.862, 0.875)
0.962 (0.860, 0.863)
0.902 (0.900, 0.903) 0.866 (0.865, 0.866)
CLOCS(CMSC)
0.736 (0.729, 0.742)
0.680 (0.649, 0.671)
0.879 (0.875, 0.880)
0.857 (0.856, 0.859)
0.905 (0.904, 0.906)
0.861 (0.861, 0.862)
PCLR
0.759 (0.757, 0. 0.762)
0.716 (0. (0.704, 0.728)
0.898 (0.8911, 0. 0.903)
0.895 (0. (0,891, 0.895)
0.927 (0.927. 0.928)
0. 0.898 (0.897, 0.899)
Our model
-EHR-ECG-Text
0.840 (0.838, 0.843) 0.859 (0.846, 0.872)
0.934 (0.930, 0. 0.887)
0.932 (0.931, 0.933)
0.942 (0.941. 0.943) 0.928 (0.928, 0.929)
Additional Evaluation Metrics
In this section, we provide the AUPRC scores for classification lasks performed on internal datasets. Table I2 shows AUPRC scores for coronary atherosclerosis, myocarditis, and cardiac amyloidosis classification tasks (an extension of Table 2 and Table 13 shows AUPRC scores for pulmonary hypertension, low LVEF, and AFib in NSR classification tasks (an extension of Table 3). We observe a similar trend in the AUPRC metric when comparing models to baseline models, did with AUROC.
22

---

Published in Transactions on Machine Learning Research (11/2023)
Table 12: An extension of Table to include AUPRC scores (mean and standard deviation over runs with different random sceds) for coronary atherosclerosis, myocarditis, and cardiac amyloidosis classification tasks: (a) linear classification, (b) fine-tuned classification. Results within 95% confidence intervals of the best result are shown in bold.
(a) Linear classification
Coronary atherosclerosis
Myocarditis
Cardiac amyloidosis
Method
10%
100%
10%
100%
10%
100%
Supervised baseline
Random Init.
0.686 + 0.013
0.768 + 0.006
0.075 + 0.019
0.239 + 0.013
0.687 + 0.009
0.795 0.004
ECG-only SSL
3KG
0.643 0.014
0.722
0.000
0.047
0.013
0.078 0.000
0.522 0.020
0.628 0.000
CLOCS(CMSC)
0.661 = 0.011
0.734 + 0.000
0.033 + 0.005
0.069
0.000
0.534 = 0.014
0.670 0.000
PCLR
0.668 + 0.008
0.761 + 0.000
0.038 + 0.010
0.082 0.001
0.618 +: 0.028
0.721
0.001
Our models
EEGR-ECG-Text
0.780 0.008
0.841 0.000
0.200 0.031
0.304
0.001
0.775 + 0.017
0.842 0.000
ECG-sEHR
0.773 = 0.018
0.843 + 0.000
0.178 1 0.028
0.291 0.001
0.777 1 0.019
0.845 0.000
ECG-Text
0.754 0.011
0.819 0.000
0.122 + 0.029
0.210 0.001
0.692 + 0.021
0.778 0.001
(b) Fine-tumed classification
Coronary atherosclerosis
Myocarditis
Cardiac anyloidosis
Method
10%
100%
10%
100%
10%
100%
Supervised baseline
Random Tnit.
0.686
0.013
0.768
0.006
0.075 0.019
0.239
0.013
0.687 0.009
0.795
0.004
ECG-only SSL
3KG
0.674 = 0.012
0.756 + 0.009
0.041 +-2 0,006
0.227
0.031
0.667 1: 0,020
0.798
0.004
CLOCS(CMSC)
0.707 1 0.008
0.747 1 0.011
0.053 1 0.018
0.255
0.015
0.693 +: 0.033
0.808
0.002
PCLR
0.735 0.010
0.763
0.005
0.043 0.009
0.257
0.026
0.750 + 0.010
0.821
0.003
Our models
:EHR-ECG-Ter
0.822 1 0.006
0.838 +: 0.007
0.248 4 0.012
0.343
0.019
0.805 1: 0.007
0.845
0.005
ECG-sEHR
0.827 0.004
0.845
0.004
0.270 + 0.012
0.349
0.013
0.812
+ 0.011
0.848
0.006
ECG-Text
0.813
0.006
0.828
0.002
0.196 1 0.017
0.302
0.008
0.779 : 0.006
0.831
0.003
E Ablation Study
We perform three ablations: (i) short ICD-10 vs. full ICD-10 codes in the sEHR modality as described in 4.2.2 using the ECG-sEHR model, (ii) report concatenation vs. entity concatenation in text modality described in 4.2.3 using the ECG-Text model, and (iii) shared space vs. fine and coarse spaces using the -CHR-ECGG-eex model as described in Section 3.2 A vocabulary of size 28593 and 42355 is used for short ICD-10 and full ICD-10 codes respectively. We show the ablation study results on the linear classification task in Table [4] The difference in performance between short ICD-10 and full ICD-10 codes is very minimal which can be attributed to the point that full ICD codes provide little to no extra information. In the report concatenation vs. entity concatenation ablation, entity concatenation yiclded better results for the majority of the diseases. We speculate that this because entity concalenation captures better long-term dependencies than report concatenation as incorporate information from one-year time window around ECGs. Note that the BEHR-ECG-TEE Text model with FAC spaces architecture yielded slightly better results than the shared space architecture.
23

---

Published in Transactions on Machine Learning Research (11/2023)
Table 13: An extension of Table to include AUPRC scores (mean and standard deviation over 5 runs with different random seeds) for pulmonary hypertension, low LVEF, and AFib im NSR classification tasks: (a) linear classification, (b) fine-tuned classification. Results within 95% confidence intervals of the best result are shown in bold.
(a) Linear classification
Polmunnry typerimmtion
Tow INFF
AFIli NSR
Method
1%
10%
100%
15%
100%
1056
100%
Superotsed bascline
Random
0.420 0.011
0,555
0.011
0.710 0.00G
0.013
0,527 0.012
0.670 0.003
0.156 00030
0.545 0.011
0.642 0.004
BCG-onnl SSL
3KG
0,383
0.022
0,512
0.008
0.537 J. 0.000
0,322 = 0.032
0.450
6.000
0.525 1. 0.000
03335 0.016
0.454 1. 0.003
0.495
0.000
CLOCS(CMSC)
0,284
0,029
0,521
0.000
0,548
0.030
0.503
0.008
0.642 0.000
0.733 00013
0.457 0.002
0.472
0.000
PCLR
0.063 0.487
0.007 0,605
0,633 1: 0.000
0.036 0.067
0.600
0.000
0,630
0.000
0.022 0.463
0.339 :: 0.003
0,895
0.000
modele
0.618
0.012
0.722
0.003
0.743 + 0.000
0.529
0.040
0.664
0.008
0.695
0.000
0.598
0.009
0.602 # 0.001
0.670
0.000
EOG-EHR
0.827
0.020
0.727
0.002
0.747 +: 0.000
0.542
0.052
0.663
0.004
0.003
0.000
0.606
0.014
0.672 +: 0.002
0.679
0.000
EGG Text
0,885
0,013
0,650
0,002
0,713 0,000
0.493
0.634
0,688 0.000
0.61 0.015
0.637 0.002
0.645
0.000
(b) Fine-tuned classification
Pulmonary aepertension
Low LVEP
APdb NSR
Mithhod
100%
10075
10%
Supereised bascline
Random Teit
0.120
0.011
0.585 0.011
0.710 +: 0.005
0.354
0.013
0:527 +: 0.012
0.039 0.003
0356 0.000
0.565 + 0011
0.02 0.001
KCG-only SST
3KG
0.429
0.015
0,600 1 0.00e
0,700 J. 0.004
0.422 = 0.013
0,582 1. 0.007
0.673 1. 0.003
0.430
00088
0.589 1. 0.004
0.690
0.004
(LOCS(CMSC)
0.497
6.014
0.714
0.011
0.002
0.071
0.006
0.189
0.001
0.038
PCLR
0.021 0,545
0,600 0.010
0,720 J. 0.004
0,586 0.004
0.002 0.649
0.692 0.002
0.013 0.509
0.585 0.006
0.063
0.002
Our models
0.868
0.012
0,723 0.006
0.746 + 0,000
0.651
0.004
0.660
0.008
0.892 + 0.008
0.620
0.021
0.651 + 0.006
0.858
0.010
ECG-EERR
0.678
0.014
0.729 :: 0.004
0.762 e: 0.004
0.643 = 0.003
0.664
0.007
0.695 :: 0.005
0.640 0.014
0.634 =: 0.009
0.661
0.003
EOG-Tood
0.604 6.010
0.731
0.008
0.638
0.078 0.001
0.585
0.008
0.653
Table 14: Results of the ablation study, Comparison of linear classification performance (AUROC, mean and standard deviation over runs with different random seeds) between short ICD codes and full ICD codes (ECG-sEHR), report concatenation and entity concatenation (ECG-Text), and shared space versus fine and coarse spaces (SEHR-ECG-Texx) Text). The best results of each ablation are shown in bold.
Coronary atherosclrrosis
Cardiac
Pulmonary hypertension
Myocarditis
Low LVEF
AFib in NSR
Method
amyloidosis
ECG-sEHR
Short ICD-10 codes
0.891
0.000
0.896 T 0.000
0.959
1 0.000
0.940
1 0.000
0.951
1 0.000
0.933 1. 0.000
Full ICD-10 cudes
0.888
0.000
0.894 1 0.000
0.958
0.000
0.940
1 0.000
0.951
1 0.000
0.934 1 0.000
ECG-Text
Entity concatenation
0.875
0.000
0.881 + 0.000
0.949
0.000
0.931 + 0.000
0.947 + 0.000
0.925
0.000
Report concatenation
0.879
0.000
0.895 + 0.001
0.939
0.000
0.921 + 0.000
0.938 + 0.000
0.925 + 0.000
-EHR-ECG-TeeR
Fine and coarse spaces
0.890 = 0.000
0.901 + 0.001
0.960 + 0.000
0.939 + 0.000
0.951 + 0.000
0.931 + 0.000
Shared space
0.883- = 0.000
0.884 + 0.001
0.955
0.000
0.936 + 0.000
0.949 = 0.000
0.928 + 0.000
24


Reviews:
[{"review_id": "99Fd9iFPt3", "reviewer": "Reviewer_V79p", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths:**\n- The proprietary dataset is very large in size.\n- The model utilizes novel modalities for ECG representation learning via contrastive learning. This is quite creative as those modalities are not commonly used together. \n- The work adopts the MultiModal Versatile Networks framework from Alayrac et al. 2020. Although it lacks methodological novelty, it can be considered a novel application of existing work.\n\n**Weaknesses:**\n- The proprietary dataset is private.\n- The code is not shared, I would highly recommend that they share their repository considering that work in this area is generally based on private models.\n- The exact contributions of the work are unclear as currently listed in the abstract. There are many results in the main manuscript, however it is also unclear which framework leads to the best improvements (ECG-sEHR vs ECG-Text vs ECG-sEHR-Text vs ECG-MTL). It currently seems that all models perform better in different scenarios, which does not really clarify the strengths of the proposed approach. \n- It's also unclear to me the difference between ECG-sEHR-Text and ECG-MTL. Does it just take the ECG encoder from ECG-sEHR-Text and apply the MTL head? Do the other models perform single task classification compared to ECG-MTL?\n- It is unclear why they conduct those specific ablation studies. It would be interesting to explore the impact of using different modalities, rather than long/short ICD and report/entity concatenation as those are not the main contributions of the work. \n- Have the authors considered how this pre-training framework could generalize to the other input modalities (not just ECG)?\n- There is no evidence of any hyper parameter tuning. This can help clarify the robustness of the proposed approaches with respect to one another.\n- There is no statistical significance testing or confidence intervals provided. They also only report the average AUROC without any standard deviation.\n- Have you considered reporting other relevant metrics? such as AUPRC\n- The notation in the methodology section is confusing. It needs to be simplified and match the notation used in the main figure. They introduce general modality-agnostic notation, and then customize it to each modality. I would suggest sticking to one. \n- The caption of the main figure should also explain all relevant notation without having to refer back to the main text. \n- The figure caption also mentions that in both ECG-sEHR and ECG-Text pre-training, the projection mapping P(es->set) is not used. Isn't it used in the latter? Where is the ECG-Text embedding space in the figure?\n- What's the motivation behind using the weighting scalars in the loss in equations 1 and 2?\n- How is the loss in equation 3 used? From my understanding, you perform ECG-sEHR pertaining (section 3.3), then ECG-Text pertaining (section 3.4), then ECG-MTL? Please clarify.\n- What's the difference between equation 4 and equation 2?\n- Can you provide a figure that clarifies the final downstream architectures for all models, including ECG-MTL?\n- Can you also provide a figure that clarifies how the modalities were linked with one another considering the varying timeframes / time differences between them?\n- Can you provide a figure that clarifies the difference between report and entity concatenation? It is unclear and the section is poorly written.\n- There are many decisions that seem to be made based on heuristics, such as keeping ICD codes that are associated with at least 50 patients.\n- Section 4.3.1 describes how the samples were labeled, can you create a table that summarizes the labeling procedure of each disease label. The text is difficult to follow. \n- The baselines are also not strong. Are there other multi-modal baselines that you can compare to? Did you perform any hyper parameter tuning for the baselines? What were the learning rates, etc.\n\n**Presentation of the paper also needs to be significantly improved:**\n- There are typos and grammar mistakes.\n- The paper is quite dense and extensive and hence the authors need to clarify certain elements of the text as described below. It mostly reads as a technical report, such that the methodological contributions / design elements are integrated with implementation details (hyper parameters, etc.)\n- It's uncommon to refer to EHR as EHRs\n- The first paragraph in the introduction is too long and it's unclear what the main message is. This also applies to other sections in the text.\n- \"weekly\" manner should be rephrased\n- consistency of capitalization in introducing abbreviations\n- The tables are referenced in random order. They also reference tables in the appendix in the main text without mentioning that they are supplementary. \n- All table and figure captions must be elaborated extensively. \n- Why is the AUROC reported as a percentage and to two decimal places? It is typically reported as a three decimal figure.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "In this paper, the authors propose a contrastive learning framework for downstream classification of ECG data. The framework includes two main components that learn similarities between ECG and structured EHR, and then ECG and textual data. \n\n**Dataset:** The structured EHR data includes ICD codes (diagnosis/procedure codes) and medication prescriptions. The unstructured data contains ECG reports, ECHO reports, pathology reports, radiology reports, microbiology reports, clinical notes and surgical notes. The authors used an internal dataset and an external dataset.\n\n**Evaluation:** The pre-training strategy is evaluated on various downstream tasks, such as classification, zero-shot retrieval, and out-of-distribution detection involving ECG as input only. They mainly use AUROC as a performance metric.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Most of my comments are summarized above. One major request would be to make the code publicly available. Even if it's a proprietary dataset, the authors can provide the code for how they implement the losses and the training procedure, which would make it easier to follow the exact training steps described in the paper. \n\nAs it stands now, the paper is not ready for publication.", "broader_impact_concerns": "The authors already discuss that the data is private and de-identified according to IRB standards.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "KN9mcDstBw", "reviewer": "Reviewer_323V", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "* I find the paper quite well written, sufficiently clearly describing the methodology followed, and the motivation/reasoning behind it.\n* I found the description of operationalization of the idea of multi-modal contrastive learning in this healthcare setting interesting, and I think it can benefit readers from this community.\n* In my understanding, the paper seems technically correct. I did not find any errors in the operation of the method or the empirical evaluation protocol. That said, I am not very familiar with all of the types of data the authors work with, so it is possible I am overlooking something. The conclusions are supported by the empirical evidence presented.\n* However, I think that the empirical evaluation of the methods should be slightly improved before acceptance (see below).\n* The authors seem to characterize the novelty of the work appropriately, although I cannot warrant it's correct, because I am not sufficiently familiar with the literature on the use of contemporary representation learning methods in the medical/ECG-modality domain.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The core value of this work lies in: 1) proposing the first instantiation of multi-modal contrastive learning framework for the purpose of learning representations of the ECG signal, 2) empirical evaluation of the proposed methods and ablating some of its domain-specific design choices.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "* In my opinion, must haves for acceptance: Given that both the training and the evaluation of the model is done using a closed-access, internal dataset, I think it particularly important for the paper to include as detailed empirical evaluation of the proposed methods as possible. Hence I would really like to see the following in the evaluation section: A) standard deviations of the results over multiple training trials (at least, over multiple fine-tuning trials with different random initializations of the linear head and random order of the data presented to the network; at best, multiple trials of SSL-based pre-training + finetuning) and bold the results for all the methods for which the standard deviations overlap considerably. B) Report the AUPRC metric as well, which seems important to me when reporting the classification performance on severely class-imbalanced data distribution.\n* Extra: My understanding is that the multimodal-SSL-initialized models (for which the results are reported in the tables) are finetuned individually on each one of the classification tasks in Table 4. I would find it interesting to add the Multitask-Learning Component on top of the SSL-pretraining, as this would allow to assess the potential of combining the multi-modal SSL representation learning with multitask learning.", "broader_impact_concerns": "No special concerns beyond the standard caution about using ML in the health domain. The authors acknowledge those appropriately.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "RCUHqpyU95", "reviewer": "Reviewer_uNfp", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The main strengths of this work are formulating an approach to combine three separate data modalities, and showing convincing evidence that it leads to better representations than existing methods. I suspect this could encourage more work on combining diverse data modalities, which are likely available in other private datasets at medical institutions around the world. I generally appreciated the evaluations, particularly the careful splitting of data and the use of external datasets.\n\nI don't see any important technical weaknesses. I ask a couple questions below, which may or may not merit some further experiments. As mentioned above, I don't find the approach very methodologically innovative, but it's reasonable and enables an interesting and challenging application of contrastive learning. One concern I would definitely like discussed is a clarification around ECG-MTL and why it's included in this paper if it's a purely supervised model.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper is about representation learning for ECGs, and the authors consider how to perform multi-modal contrastive learning using a private dataset combining ECGs, structured EHRs (information about diagnoses, prescriptions, tests, etc) and unstructured EHRs (free-form text). It differs from previous work on self-supervised and contrastive learning as follows:\n\n- SSL and contrastive learning are performed mainly in a uni-modal way for general domain images, or for multi-modal datasets combining text and images (e.g., captioned images scraped from the internet). Most applications of SSL/CL to medical data are also either uni-modal with images, or with text-image pairs. This work considers three modalities: ECG waveforms (time series), sEHRs (not quite text), and free-form text from unstructured EHRs.\n- SSL and contrastive learning have been previously applied to ECGs but typically in a uni-modal fashion. One work combined ECGs with a different modality (labs and vitals time series), but the use of EHR data here seems novel.\n\nThe recipe for contrastive learning is usually straightforward, but the authors confront several difficulties here. One is determining the best encoders for each modality: they use a CNN for ECGs (a standard choice), a pre-trained LM for the unstructured EHR data (GatorTron), and a custom BERT-like model for the sEHR data. They must also consider how to compare between three modalities: as shown in Figure 1, there are separate embedding spaces in which two contrastive losses are calculated (ECG-sEHR, and ECG-sEHR-Text). This seems reasonable, and the loss functions (eqs. 1 and 2) are the standard losses for each space. Methodologically, this work is not too novel, but the choices are sensible and not straightforward to execute on (particularly training the custom BERT-like model for sEHR data).\n\nThe authors in fact train three separate multi-modal contrastive learning models, which utilize either type of EHR or both types. In addition, they train ECG-MTL, a multi-task model trained in a supervised manner for a large number of classification and regression tasks. (Actually, I could not understand whether this was a separate model, or fine-tuned from one of the multi-modal models - I request clarification on this below.)\n\nThe evaluations show that the new models are generally more effective than existing contrastive learning methods for a variety of downstream tasks, when tested with linear probing or full fine-tuning, and when tested on either the internal test set or external open-source datasets. Due to the strong existing backbones, the new models are also more sample-efficient. \n\nThe authors also show results for zero-shot retrieval tasks, and out-of-distribution detection, but while interesting, these seem less clinically relevant.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "A couple thoughts and questions:\n\n- The results in Tables 2-4 are generally quite negative regarding the existing contrastive learning methods. They're rarely competitive with the new models (which is perhaps a good thing), but they're often worse than a simple supervised baseline. This is perhaps not a direct contradiction of previous work, as they did not use the same datasets, but it does suggest that those are ineffective representation learning methods. This is odd, because they use much more data and rely on sound contrastive learning principles. Can the authors comment on this, and perhaps make room in the paper to discuss why those methods don't work?\n\n- The ImageNet comparison in contribution #3 was confusing to me. ImageNet is an open-source dataset, this is a model trained on a closed-source dataset. Can you rephrase this contribution so the comparison is more clear, or perhaps omit the comparison?\n\n- As mentioned above, I could not tell whether ECG-MTL was trained in a purely supervised fashion or fine-tuned based on a backbone from the previous multi-modal models. If it's the former, I'm confused why it belongs in this paper. If it's the latter, I would have expected more discussion of this point in Section 3.5, and perhaps a comparison between backbones from which it's fine-tuned.\n\n- In Section 3.2, the claim about the modalities having different levels of granularity is interesting, but this isn't accompanied by any empirical evidence or ablations. If the granularities were more similar, how would the learning approach differ? And could it be added as an ablation study? It's not that the results are strong as is, but from a scientific perspective, omitting any empirical evidence on this point is unsatisfying for interested readers.\n\n- \"GatorTron\" is misspelled in Figure 1.\n\n- Just a nit, but assuming L2 normalization when computing cosine similarity is unnecessary. The standard definition of cosine similarity performs the normalization internally ([PyTorch](https://pytorch.org/docs/stable/generated/torch.nn.CosineSimilarity.html), [Wikipedia](https://en.wikipedia.org/wiki/Cosine_similarity)).\n\n- I couldn't tell why eq. 2 is repeated on the very next page in eq. 4.\n\n- For context, can the authors provide some information about SOTA results on the external datasets? That information is missing, and it would be interesting to know whether this method enables better performance.\n\n- The explanations about IRB approval don't seem to belong in the conclusion. Perhaps they should be moved earlier, for example to a section where the datasets are introduced?", "broader_impact_concerns": "I have no broader impact concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors use existing techniques on a new dataset, and analyze performance results on both internal and external benchmarks.  Their analysis yields strong evidence that their multimodal approach can improve upon existing SSL-for-ECG techniques for producing predictors in low samples and match (or surpass) supervised baselines in larger data regimes.  While it is disappointing that the dataset that leads to these performance improvements is closed, it is understandable given the nature of the dataset. I believe these results will find an audience, perhaps, e.g., other researchers at health centers with access to their own ECGs and CHRs.  I definitely encourage following up with code for to aid in the reproduction of these results by such researchers.  Reviewer concerns were, for the most part, addressed.
Decision: Accept (poster)