ECG Semantic Integrator (ESI): A Foundation ECG Model Pretrained with LLM-Enhanced Cardiological Text
Authors: Han Yu
Peikun Guo
Akane Sano
Venue: Accepted by TMLR ()

Abstract: The utilization of deep learning on electrocardiogram (ECG) analysis has brought the advanced accuracy and efficiency of cardiac healthcare diagnostics. In this work, we address a critical challenge in the field of ECG analysis with deep learning: learning robust representation without large-scale labeled datasets. We propose ECG Semantic Integrator (ESI), a novel multimodal contrastive pretraining framework that jointly learns from ECG signals and associated textual descriptions. ESI employs a dual objective function that comprises a contrastive loss and a captioning loss to develop representations of ECG data. To create a sufficiently large and diverse training dataset, we develop a retrieval-augmented generation (RAG)-based Large Language Model (LLM) pipeline, called Cardio Query Assistant (CQA). This pipeline is designed to generate detailed textual descriptions for ECGs from diverse databases. The generated text includes information about demographics and waveform patterns. This approach enables us to compile a large-scale multimodal dataset with over 660,000 ECG-text pairs for pretraining ESI, which then learns robust and generalizable representations of 12-lead ECG. We validate our approach through various downstream tasks, including arrhythmia detection and ECG-based subject identification. Our experimental results demonstrate substantial improvements over strong baselines in these tasks. These baselines encompass supervised and self-supervised learning methods, as well as prior multimodal pretraining approaches. Our work shows the potential of combining multimodal pretraining to improve the analysis of ECG signals.

Published in Transactions on Machine Learning Research (10/2024)
ECG Semantic Integrator (ESI): A Foundation ECG Model Pretrained with LLM-Enhanced Cardiological Text
Han Yu
ky29@rice.co edu
Deportment of Flectrionl and Computer Engineering
Rice University
Peikun Guo
Pg34 Orice.
Department of Compuler Science Rice University
Akane Sano
akane.sano@rice Cricee.wwwri..oo edu
Department of Electrical and Computer Engineering
Rice University
Reviewed on OpenReview: https: // opwreeveee.........mm net/ forum? id- cKKhef
Abstract
The utilization of deep learning on electrocardiogram (ECG) analysis has brought. the ad- vanced accuracy and efficiency of cardiac healthcare diagnostics. In this work, address critical challenge in the field of ECG analysis with deep learning: learning robust represen- tation without large-scale labeled datasets. We propose ECG Semantic Integrator (ESI), novel multimodal contrastive pretraining framework that jointly learns from ECG signals and associated textual descriptions. ESI employs a dual objective function that comprises a contrastive loss and a captioning loss to develop representations of ECG data. To create a sufficiently large and diverse training dataset, we develop a retriivallauurrennee genera- tion (RAG)-basod Large Language Model (LLM) pipeline, called Cardio Query Assistant (CQA). This pipeline is designed to generate detailed textual descriptions for ECGs from diverse databases. The generated text includes information about demographics and wave- form patterns. This approach enables us to compile a large-scale multimodal dataset with over 660,000 ECG-text pairs for pretraining ESI, which then learns robust and generaliz- able representations 12-lead ECG. We validate our approach through various downstream tasks, inchuding arrbythmia detection and ECG-based subject identification. Our experi- mental results demonstrate substantial improvements over strong baselines these tasks. These baselines encompass supervised and self-supervised learning methods, as well as prior multimodal pretraining approaches. Our work shows the potential of combining multimodal pretraining to improve the analysis of ECG signals.
The training code and generated waveform descriptions available at https: htgp::///itte
com/ ht/p//omp--ae wocop-wwlll--aeeeeeeeeeeeeeellle org/
1 Introduction
The electrocardiogram (ECG), which provides non-invasive and comprehensive view of the heart's 's electrical activity, is an important tool in cardiovascular diagnostics and clinical Maaiiionmmmiing/Kgiiiitit et al. (007)). For example, ECG has been extensively used in various clinical scenarios, such as diagnosing cardiovascular diseases (Jain et al. 2014), obstructive sleep apnea (Faust et al. 2016), and Parkinson's 's disease (Haapaniemi ct al. 22001), etc. On the other hand, the rapid development deep learning has triggered general interest ECG data analysis using data-driven approaches. These deep learning methods, recognized for their ability to learn complex representations, have been proven highly effective in enhancing the accuracy and predictive

---

Published in Transactions on Machine Learning Research (10/2024)
capability of ECG analysis (Hanmun et al. 2019 Typically, the initial step in utilizing ECG signals involves extracting features from the raw data, either through conventional feature engineering or more recent deep
learning backbones, such as ID convolutional neural network (CNN) Zhu et al. (2020); Balogiu et al. (2019):
Jing et al. (2021) and Transformer models Meng et al.] ) Behinacin et al. (2021); Natarajan et al.
(2020g; Guan et al. (2021]; Yan et al. (2019). However, these supervised methods oflen require large-scale and high-quality annotated training samples, which are costly to obtain (Mincholé & Rodriguez 2019).
To reduce the reliance on extensive annotations, researchers have explored self-supervised learning (SSL) techniques for ECG signals (Eldele et al., 2021; Kiyasseh et al., 2021; Yu et al. 2023 Gopal et al. 2021 These methods utilize the unlabeled data when pretraining deep feature extractors. Nevertheless, the SSL strategies, which include tasks such as aligning different signal views or reconstructing masked segments, mainly focus on signals. This means these SSL methods pay attention mainly to the waveform characteristics and neglect the semantic meanings of the signals. Consequently, there is no guarantee that those methods can effectively Iearn robust representations during the protraining phase to enhance the ECG analysis in the downstream lasks.
Other studies leverage multimodal learning approaches and incorporate additional modalities, such as de- scriptive text, into the pretraining process. This approach has shown excellent pretraining performance by enabling a more nuanced and comprehensive understanding of the data (Radford et al. 2021 Jia et al. 2021 Yu et al. 2022b). Motivated by the success of multimodal pretraining on image-text pairs, researchers have developed similar methods for ECGs paired with other modalities including label text (Li et al.) 2024), elec- tronic health records (EHR) [Lalam et al. 2023), and clinical reports (Liu al.) 2024). However, acquiring such modalities in large quantities for ECG can also be costly, as ECG analysis would require expertise- depended semantic information compared to general computer vision tasks. Additionally, the variability in terminology and detail across different ECG datasets and sources causes challenge when combining multiple data sources for a larger scale of pretraining data.
To address these challenges, we introduce a two-step multimodal contrastive pretraining framework to en- hance the representations learned from ECG signals. We propose a retrieval-augmenteed generation (RAG)- based pipeline, Cardio Query Assistant (CQA), to generate standardized and enriched textual descriptions for ECGs. By leveraging the capability of RAG to retrieve relevant information from ECG textbooks, CQA transforms basic ECG conditions into enhanced text descriptions that include palient demographics and spe- cific waveform characteristics. Further, based on the enriched textual descriptions from CQA, we introduce ECG Semantics Integrator (ESI), a contrastive learning framework with a captioning loss inspired by (Yul it al., 2022b). ESI aligns ECG signals with their corresponding text annotations, which aims to pretrain the encoders for an enhanced semantic understanding of F.GG content. Our contributions are summarized as follows:
We introduce RAG-based ECG description generation pipeline CQA that constructs descriptive textual context for ECG samples using demographic information and diagnostic conditions.
We develop an ESI framework with both contrastive and captioning capability in pretraining train an ECG foundation model on approximately 650,000 12-lead ECG signals.
Compared to strong baselines including the prior SOTA supervised and SSL methods, our evaluation demonstrates promising performances in arrbythmia detection and ECG-based user identification. For instance, we observe an improvement of 1.6% in AUC scores for diagnosing arrhythmia classes, and 3.5% improvement in identifying subjects when compared to prior SSL methods.
2 Related Work
2.1
Multimodal Representation Learning
Recent studics have introduced foundation models for integrating image and text, which involve both visual and vasion-language pretraining. Vision-language pretraining aims to learn multimodal foundation models with enhanced performance on downstream vision and language tasks. Models such CL.I RRdford et al.
2

---

Published in Transactions on Machine Learning Research (10/2024)
2021 ALIGN (Jia et al. 2021), ), Florence (Yuan al. 2021 and LiT (hai et al. 2022) use dual-encoder architectures where image-text pairs are mapped into shared embedding space. These models are trained using contrastive objective that pulls together embeddings of matched image-text pains while pushing apart those unmatched pairs. This approach has proven effective in developing robust and transferable representations for both image and lext modalities, which siguificantly improves the performance on various vision-language tasks.
In parallel, other research has explored encoder-decoder architectures that leverage generative objectives. Models such as CoCa (Yu et al. 2022b), SimVLM (Wang et al. 2021). BLIP (Li et al 2022). BLIP-2 (Li ct al. 2023b) and OFA (Wang et al., 2022) integrate both encoding and decoding processos, which enables the model to generate text from visual inputs. These models use language modeling alongside image-text matching and generate semantically meaningful outputs. This line of study also demonstrates a flexible and powerful potential for multimodal learning and demonstrates strong performance across a variety of vision-language benchmarks.
The power of these multimodal pretraining techniques has not been limited to general image and text data. Researchers have adapted these approaches to medical imaging, particularly in radiography paired with clinical reports {Liu et al.) 2023 You et al. 2023 Wan et al., (024).. However, despite the progress in image-based applications, applying these multimodal pretraining methods to FCG signal processing remains relatively underexplored. In this study, we extend these advanced techniques to ECG data for more robust and generalizable representations in clinical settings.
2.2
ECG Diagnosis with Deep Learning
2.2.1
Supervised Methods
Deep learning applications in ECG diagnosis have drawn significant attention Liu et al. 2021 Pyakillya
et al. 2017 Sannino & De Pietro 2018 Wagner et al., 2020: Smigiel et al. 2021 Mostafa et al. 2019 For instance, Smigiel al. (2021) proposed CNN model with additional entropy- fcatures for arrhythmia classification. Their method achieves an AUC score 0.91 across five classes. Mostala el. al. 2019)) conducted comprehensive review of deep learning applications in ECG analysis for sleep apnea detection. They highlighted the success of models such as CNNs and recurrent neural networks (RNNs), which achieve over 90% accuracy on specialized datasets. Despite the proven effectiveness of these methods, the acquisition of clinical annotations required for these methods is often expensive.
2.2.2 Unimodal Representation Learning in ECG
Given the expensive nature of clinical annotations, there has been a growing interest in pretraining methods designed to reduce reliance labeled F.G sequences (Sarkar & Etemad, 2020 Mehari & Strodthoff] 2022:
Oh ct al. 2022). For example, Mchari & Strodthoff (2022) applied well-known SSL frameworks such SimCL (Chen et al. 2020), BYOL (Grill et al. 2020), and CPC (Oord et al. 2018) to pretrain models on 12-lead ECG data. These models showed enhanced robustness, reflected in 2% increase in AUC score for 5-class arrhythmia classification compared to purely supervised models. However, even though pretraining strategies generally provide insights into performance improvement and decrease reliance on labeled data, these methods are often limited by their focus solely on signal waveforms. The emphasis on waveform alone does not ensure the capture of clinically relevant semantic information. As result, multimodal approach incorporating both ECG waveforms and corrosponding clinical text helps acquire more meaningful and transferable ECG representations for various downnstrea tasks.
2.2.3 Multimodal Representation Learning in ECG
Although few, some studies have begun to explore the alignment of ECG signals with other modalities such as textual descriptions, EHR, and clinical notes (Li et al., 2023a Lalam et al. 2023 Liu et al. 2024). For instance, Lalam al. (2023) utilized identical encoders to extract and contrastively align emboddings from ECG, EHR, and clinical notes. The pretrained model showed promising results in clinical diagnosis. Lin al.
h adopted similar approach to couple F.G signals with clinical notes and enhanced the ECG encoder's
3

---

Published in Transactions on Machine Learning Research (10/2024)
Database Construction
UPU
Chunking
Text
This ECG signal associates with (SAGE) year old (SGENDER).
segment
Vector
224 3.3 1.9...
Domain-speciie
knowledge documents
Vector Database
The cardiac condition is (RBBE), with potential
Training-Fri FFre QA
"How
Embedding
Look-up
Answer -
(RBBB)
Vector
reflected
(*PProlonge QRS duration.
waveform: (SWAVEFORM
ECG Waveform?"
chroma
RSR pattern V1-V3 (M-
ECG
shaped))
Description
Figure The Cardio Query Assistant (CQA) Framework employs novel knowledge- based approach generate detailed and clinically relevant textual descriptions for ECG signals, which translates ECG condi- tions into enriched ECG waveform patterns.
effectiveness in wero-shot arrhythmia detection. However, the pretraining processes of these methods depend costly annotations such as climical notes and EHR, which are challenging to acquire on a large scale, Morcover, the variability textual descriptions or reports associated with ECGs due to differences in detail, terminology, and style among clinicians and clinical contexts may complicate the learning of consistent mappings between ECG signals and text. The consequent variability could lead to misalignments between the ECG-text pairs. In this study, we propose to utilize retrieval-augnnented generation (RAG)- pipeline to construct contextual ECG textual data without relying on costly notes and EHRs. Additionally, we introduce captioning task in our model to achieve more nuanced representations.
3 Methods
In this section, we introduce our approach in three main components: (1) RAG-based ECG description pipeline, Cardio Query Assistant (CQA) and (2) the contrastive captioning pretraining framework, ECG Semantics Integrator (ESI).
3.1 Cardio Query Assistant (CQA) Framework
The Cardio Query Assistant (CQA) Framework, shown in Figure I is designed transform ECG con- dition labels into detailed descriptive text. The generated text incorporates demographic information, ECG conditions, and enriched waveform details. An example of a raw ECG signal, its associated metadata, and The developed CQA outlined as follows:
the generated FCG waveform description is provided in Figure 2
3.1.1 Establishing a Domain-Specific Knowledge Database
To leverage the enhanced interpretation of ECG conditions with domain expertise, we develop a comprehen- sive vector database from sppeiff literature of authoritative medical texts guided by two textbooks: (1) ECG Workout: Exercises In Arrbythmia Interpretation by Huff (2006). and (2) 12-Lead ECG: The Art of Interpretation by Garcia (2015). To extract and encode this information into a usable format, we employ the leal-embeadding-ullee 002 APT (OpenAT] 2023) because of its efficiency and performance. The resulting embeddings are then systematically organized using the Chroma database management tool, which was chosen for its robustness and ease of integration with the LangChain Python library (Mendable) 2023
3.1.2 Enhancement of ECG Semantics
The CQA Framework enriches the ECG-associated information through a comprehensive retrieval-angmenteed process. With the pro-constructed domain-knowledge database, the RAG-based approach enables CQA query related knowledge using given information such as standard clinical labels, standard commuications protocol for computer-assisted ECG (SCP) statements, diagnostic interpretations, and machine-generated
4

---

Published in Transactions on Machine Learning Research (10/2024)
12-Lead ECG Signal
Patient
Age
Gender
XXX
47
M
ECG Metadata
Sinus rhythm, Abnormal R-wave progression, early transition, ventricular hypertrophy, Nonspecilic sbnoomaliiiss, diffuse leads, Prolonged QT intervel
Conditions
This patient is 47-year-old Male.
ECG Findings:
Sinus rhythm, Abnormal R-wave progression, early transition, Left ventricular hypertrophy, Nonspecific abnormalites, diffuse leads, Prolonged QT interval.
0.0
Potential ECG waveforms:
Normal sinus rhythm has regular rhythm with rate of 100 bests per minute. The P waves are normal in size, shape, and direction, positive lead Il. One P wave precedes each QRS complex. The PR interval is normal, between 0.12 to 0.20 seconds. The QRS complex is normal. 0.
9
less duration.
Rowave progression early transition, the R wave becomes smaller diminishes the precordial leads from V1 to V8. The R wave is initially tall in V1 but progressively precordial leads, becoming smaller than the S wave earlier than expected, typically by V4.......
condition
3
amplitude across the
9
0.0
ECG Metadata
2000
Time (sampies)
Figure 2: Example of 12-lead ECG signal and its associated metadata. The left. side displays the 12- lead ECG waveform, which provides visual representation of the heart's electrical activity. The right side includes relevant patient demographic information (age, gender), sunmary of ECG metadata including high-level clinical findings (e.g., Sinus rhythm, Abnormal R-wave progression), and a detailed textual de- scription of the ECG conditions generated with the CQA pipeline.
reports associated with ECG data. SCP statements are standardized lextual formats that provide consistent documentation of ECG findings, following international protocols for computer-assisted ECG interpretation. While these datasets provide valuable diagnostic information, they often lack explicit details about the waveform patterns that are critical for thorough ECG analysis.
To address this gap, the CQA Framework employs RAG approach that enables it to query the knowledge database, which is introduced in the prior section, for relevant information and generate comprehensive textual descriptions of the potential waveform characteristics. For instance, SCP statements and standard arrhythmia diagnoses may not directly include detailed waveform descriptions; however, by querying the komain-knowlodgo database, the CQA Framework can synthesize this enriched information, producing de- tailed descriptions of the waveform patterns that correspond to specific ECG conditions.
With the constructed RAG pipeline in CQA, we asked the waveform information by using simple prompts of 'How is ECG Condition reflected in 12-lead ECG'? The output of the CQA framework is organized utilizing an LLM, e-g., GPT-3.5, to generate the potential waveform details from the queried knowledge base. Take an example the cardiac condition of the Right Bundle Branch Block (RBBB), By executing targeted queries in our database, retrieves and generates descriptive context for specific waveform attributes. For example, the ECG-associated information of "RBBB" is queried and converted into related waveform features, including "prolonged QRS "uration" and "M-shaped RSR' pattern in leads V1-V3."
5

---

Published in Transactions on Machine Learning Research (10/2024)
This patient year-old female...
... was diagnosed with RBBB...
Encoder
waveform information:
QRS duration 120ms RSR' pattern in v1-3
("M-shaped" QRS complex)
T1
T2
T3
TN
Cross- Modality Decoder
Textual Prompts
Captioning Loss
S
81 T2
S1 T3
S1 Ty
S2
S2 T
S2 T
S2
Sequence Encoder
Ss
T
S$ T
Sq
ECG Signal
SN
S Sw Sy
Sx Ty
Contrastive Loss
Figure 3: The ECG Semantics Integrator (ESI) is built based on ECG signal encoder with text encoder using captioning and contrastive losses for unified representations. This architecture learns from the align- ments between detailed textual prompts and the corresponding ECG waveform data, which aims to capture nuanced clinical insights for enhanced diagnostic tasks.
3.2 Multimodal Contrastive Captioning with ECG Semantics Integrator (ESI) Framework
The ESI framework aims improve the quality of representation extracted from ECG signals by pretraining specialized ECG encoder alongside textual encoder. This dual-modality training method has been proven by cutting-edg' studies in contrastive language-image pretraining (CLIP) (Radford et al. 2021] and CoCa (Yu et al. 2022b) methodology. Inspired by these two studies, we developed our pretraining architecture with both contrastive and generative objectives. While we recognize there are newer approaches in multimodal contrastive learning beyond CLIP and CoCa as discussed in Scction 2.1 the choice development can be motivated by several factors. First,, the CLIP structure well-proven by multiple prior works, including pretraining language with medical images (Liu
2023 You al., 2023 Wan et al., (2024) and physio- logical signals (Li et al. 2023a Lalam al.) 2023 Liu et al. 2024). The core contrastive objective of CLIP is well-suited for our primary goal of aligning ECG signals with their textual interpretations. Moreover, the relative simplicity of the CLIP architecture allows for seamless integration of our captioning loss inspired by CoCa YYu et al. (022bb). We further validate the contribution of the contrastive and generative objective via ablation study as shown in Section 5.1
For the ECG encoder, we have chosen one-dimensional modified version of the ConvNext architecture considering the sequential nature of ECG waveform data (Woo al. 2023), which has been proven for its capacities of extracting both local and global contexts with the designed convolutional kernels. In parallel, the textual encoder utilizes BiRR,,,BERT, derivative of the BERT architecture pretrained on biomedical texts, to effectively embed medical terminologies (Yasunaga et al. 2022). BioLinkBERT is an extension of the standard BERT model DDevlin et al. 2018), specifically designed to improve the understanding of biomedical texts. Unlike traditional BERT, which processes each document independently, BioLinkBERT is pretrained on biomedical litcrature from PubMed, which takes advantage of the natural links between documents such citations and references. This pretraining strategy motivates us to leverage BioLinkBert for tasks that require deep understanding of biomedical concepts and terminology.

---

Published in Transactions on Machine Learning Research (10/2024)
3.2.1 Multimodal Contrastive Learning
Inspired by the previous vision language pretraining approaches (Radford et al. 2021 Yu et al. 2022by, our framework uses two pretraining objectives for comprehensive learning, including contrastive loss for robust representation learning and captioning loss for semantic alignment.
We employ the dual-encoder contrastive learning framework following the prior studies. Compared to pretraining with single-encoder as signal-focused frameworks, e-g,, SimCTR (Chen et al. 2020) and BYOL (Grill et al.) 2020), the dual-encoder approach in this study leverages the semantic information from the textual modality. Both encoders aim to project the inputting ECG and text into a unífied embedding space. Consequently, the two encoders are jointly optimized by contrasting the paired text against others the sampled batch:
Contrastive Loss:
1 N 2 log
exp (1/
exp (T: Sifo) S//0)
Coo
Σ log
ブ exp (5,
j==1 exp (TTS)/O)
ecg-to-text
text-to-ecg
with S: and T; representing the normalized embeddings from the ECG signal and lext encoders for the i-th ECG-text pair, N is the batch size during training, and a as the temperature scaling factor. This dual-encoder approach has been working promisingly on enabling cross-modal alignment applications such as zero-shot classification (Radford al. 2021 Yu et al., 2022b).
Captioning Loss: While the dual-encoder approach encodes the text as embedding for the contrastive learning purpose, the generative approach aims for detailed granularity and requires the model to predict the exact tokenized texts with ECG and preceding texts. This approach encourages the encoders to capture the semantic information embedded in the texts actively. Inspired by the image-text multimodal pretraining study CoCa (Yu et al. 2022b), we design to align the generated textual descriptions with the corresponding ECG signals by additionally defining a captioning loss Loap similar to that used in image captioning lasks (Vinyals et al. 20155):
Capp = Σ log P(tilte,,
where 4 represents the i-th token in the textual description, tei denotes all the preceding tokens, S is the ECG signal, and 9 represents the parameters of both encoders and the cross-modality decoder.
The overall pretraining objective the combination of both contrastive loss and captioning loss, denoted as:
L= DCon Loon XCop
CCup
where Acon and XCap are the loss weighting hyperparameters for the introduced objectives. We set these two weighting parameters equally to 1 in this study. By jointly optimizing these losses, the ESI Framework aims to learn a multimodal representation that enriches the semantic link between ECG waveforms and their textual explanations. This method anticipated to improve performances in downstream tasks that leverage the waveform details and demographics, such diagnosing arrhythmia and performing large-scale patient identification using ECG data.
4 Evaluation
In this section, we describe evaluation settings and experimental results. We first introduce the informa- tion on the datasets used and tasks performed in this study, along with the baseline methods we used in the comparisons. Our experiments explore three settings: zero-shot lcarning, linear probing (frozen features), and fine-tuning. In our experiments, we conduct multiple resampling runs to assess the robustness of the model, particularly for the linear probing and fine-tuning settings. Given that the encoder is frozen during lincar probing evaluations, the randomness is only applicd the output lincar heads during the linear prob- ing setup. For each setting, we average the results over five different runs and report the standard deviations alongside the mean performance metrics.

---

Published in Transactions on Machine Learning Research (10/2024)
Table Summary of datasets used in the pretraining stage
Dataset
Duration
Sampling Rate # of Training Samples Associated Information
PTB-XL
10 seconds
500 Hz
17K
Demographics, SCP Code
Saapmaan-Shhxxigg
10 seconds
500 Hz
45K
Demographics, Arrhyth- mia Diagnosis
MIMIC-IV-EGG
10 seconds
500 Hz
600K
Demographics, Machine- generated ECG Reports
4.1 Training Setup
Pretraining Datasets. The proposed ESI signal encoder is pretrained from scratch. Therefore, the pre- training datasel directly impacts the model'ss generalizability. We constructed large pretraining set combin- ing three large-scale datasets with over 650,000 ECG-text training pairs. These datasets covers Chapman- Shaoxing (Zheng et al. 2020) PTB-XL (Wagner et al., 2020), and MIMIC-ECG (Gow et al. 2023 Each dataset contains 12-lead and 10-second ECG recordings sampled at 500 Hz. Here detailed breakdown of each dataset:
PTB-XL: This dataset consists of 21,837 12-lead, 10-second ECG recordings from 18,885 partici- pants. We followed the training and test data split guidelines outlined in the original publication (Wagner et al.] 2020) and only used the training samples (17k) in the pretraining task. These samples include demographic data and SCP codes.
Chapman-Shaoxing: This dataset offers larger set of 45k samples with associated demographic information and arrhythmia diagnoses.
MIMIC-IV-ECG: This is the most extensive dataset with 600k samples accompanied by demograph- ics and machino-generated ECG reports.
The variety and volune of data provide a comprehensive foundation for the pretraining of models. Table ! summarizes the overview information of each dataset used in pretraining.
Implementation. During the pretraining phase of the ESI model, we make specific choices regarding the encoder architectures, optimizer, learning rate scheduler, training hardware, and batch size. The ECG sigual encoder within EST utilizes 1D ConvNeXt-base (Woo et al. 2023) backbone as the default. architecture. This choice allows the model effectively capture the spatial features within the ECG signal data. For text encoding, leverage BioLinkBert (Yasunaga et al.) 2022) the default due to its proven capabilities in handling biomedical text data. The AdamW optimizer is employed for optimization during the pretraining process. We opted for an initial learning rate of 5 x 10-5 to facilitate efficient convergence. To further adjust the learning rate throughout training, a warm-up phase of 5 epochs out of the total 30 epochs is implemented. This warm-up phase allows the model to gradually adjust to the training data before applying the main learning rate. Additionally, a learning rate decay of 0.1 introduced after every epochs prevent overfitting in the later stages of training.
The pretraining process is conducted on server equipped with 4 Nvidia A100 GPUs. This hardware configuration provides the computational resources necessary to handle the large datascts used for pretraining efficiently. To leverage the capabilities of these GPUs effectively, a batch size of 48 samples is used on each GPU during training.
In addition the main EST model, we implement parametor-cfficient variant named ESI-tiny. This variant utilizes a ConvNeXt-tiny architecture the ECG encoder, which pretrains a model with a smaller overall size. This can be beneficial in scenarios where computational resources are limited.
8

---

Published in Transactions on Machine Learning Research (10/2024)
PTB-XL Arrhythmia (AUC)
Supervised fognal-focused SSL ESI
ICBEB
0.935
ICBEB Arrhythmia (AUC)
Identification (ACC)
0.961
0.80
0.978
0.35
0,830
0.9R5
0.70
0:65
0.92
0.92
0.45
0.94
0. 0.64
0.96
0.50
0.98
0.98
1.00
ICBEB Identification (AUC)
PTB-XL Identiiiction (AUC)
0.85
PTB-XL
Identification (ACC)
Figure 4: Comparison of the proposed ECG Semantics Integrator (ESI) with the best performances from bascline methods including the supervised models and signal-focused self-supervised learning (SSL) pre- trained models. Compared to the baselines, ESI is multimodal contrastive pretraining framework that leverages both ECG signals and corresponding textual descriptions to learn enhanced ECG representations. The evaluations of arrhythmia diagnosis and identification are conducted on datasets including PTB-XL and ICBEB, with metrics of area under the ROC curve (AUC) and accuracy (ACC).
4.2 ECG Semantics Integrator (ESI) For Downstream Tasks
After the pretraining stage, the encoder can be applied three different manners, including zero-shot inference, linear probing, and fine-tuning on various downstream lasks. The aim to validate the robustness of the learned representations and the practical utility of the model in real-world clinical settings. While directly assess the representations of the learned framework,
the zero-shot inference and lincar probing the fine-tuned models usually provide the best performances among these three methods, with updating parameters through downstream tasks. As shown in Figure ! the fine-tuned encoder from our ESI method outperforms the supervised and self-supervised baselines for different downstream tasks.
4.2.1 Zero-Shot Evaluation
Zero-shot evaluation assesses the model's capacity to understand and infer information from ECG signals without any task-specific fine-tuning. Following the definition used in the context of CoCa. (Yu et al. 2022b) and CLIP (Radford al.) 2021), our zero-shot evaluation strategy ensures that while the model has been exposed to vast array of ECG and text pairs during pretraining, it has not seen any supervised examples from the downstream tasks. In the ESI Framework, each ECG signal's embedding is compared against range of possible textual labels for different cardiac conditions without task-specific fine-tuning. The model selects the textual description that has the closest embedding distance to the ECG signal's embedding, which determined by a similarity metric of cosine similarity. This process demonstrates the model's understanding ECG data and its ability correlate with accurate clinical descriptions directly after pretraining, which aims to demonstrate the potential generalizability of the learned representations without fine-tuning the specific tasks.

---

Published in Transactions on Machine Learning Research (10/2024)
4.2.2 Linear Probing
Linear probing a strategy that makes use of the representations learned by the EST Framework's encoders. In this sctup; linear classifier is utilized on top of the frozen encoders for different downstream tasks such as arrhythmia delection and palient identification. As the only trainable part of the model is the classifier head, thus, the quality and robustness of the representations from the pretrained encoder play essential role in the linear probing strategy.
4.2.3 Fine- Tuning
To introduce more flexibility into the pretrained signal encoder, we can also fine-tune the entire framework on a set downstream tasks. Similar to the linear probing strategy but with trainable encoders, this fine- tun-ttnnig strategy aims to explore the full effectiveness of the structure with pretrained parameters its initialization for reamstream lasks.
4.3 Downstream Task: Arrhythmia Diagnosis
Cardiac arrhythmias are a significant contributor to cardiovascular diseases, and there is a demand for accurate and reliable detection methods for clinical use. We evaluated our proposed method on arrhythmia detection using two datasets, PTB-XL (Wagner et al.] (2200) and ICBEB (Liu et al. 2018). As described in Section 1.1] we follow the training and test data split guidelines outlined in the original PTB-XL publication (Wagner et al. 2020) to divide the PTB-XL dataset. The training set is used for fine-tuning the model and the test set, which is not seen during the pretraining, is used for evaluation. The ICBEB dataset, which is not used during the pretraining, consists of 9,831 12-lead ECG signals from 9,458 patients (Liu et al. 2018). We adopt the processing settings from a prior benchmark study (Strodthoff et al. 2020), which results in 6,877 training samples and 2,954 test samples. Based on this configuration, we evaluate the model's effectivencss across three settings, including fine-tuning, linear probing, and zero-shot learning.
4.3.1 Fine-tuning & Linear Probing
To perform comprchensive evaluation, we compare our proposed method with specialized supervised meth- ods, including long short-term memory (LSTM), XeesNe1101, ResNet50, ensemble methods implemented fusion network (MLBF-Ne.
in an ECG benchmark study (Strodthoff et al. 2020 a multi-lead-brant (Zhang et al. )021), and a multi-view multi-scale neural network (MVMSN) (Yang et al. 2023 Besides the supervised learning methods, we also cover the comparison between our method and aggna-foccsed SSL methods including SimCLR (Chen et al. (020), BYOL (Grill et al., 2020), CLOCS (Kiyasseh el al. 2021] and LEAVES (Yu et al., 2022a). The deep learning backbone used for training these methods is ConvNeXt- base, the same the proposed ESI, to ensure fair comparisons. We perform both the lincar probing and fine-tuning strategies for those pretrained methods to assess the quality of learned representations during the pretraining phase. Additionally, we benchmark against MERT. (Liu et al. 2024) under a frozen encoder setting as presented in their study.
Tablee]shone the performances of the evaluated methods. In the supervised learning methods, the ConvNeet and ConvNeXt-tiny encoders show lower performance compared to specialized methods such as -Let and MVMSN. Conv.NeXt-tiny outperforms the larger ConvNeXt model, potentially due to overfitting with the smaller dataset. After pretraining, the ConvNeXt-based ESI method achieves the best performance both PTB-XL and ICBEB datasets under both the frozen encoder and fine-tuning settings. Notably, the mul- timodal pretraining methods (EST and MERL) significantly outperformed the signal-focused methods such as SimCLR and BYOL. This supports our hypothesis that signal-focused approaches may have limitations in learning robust and transferable representations for downstream tasks compared to the multimodal pre- training methods. The superior performance of the pretrained ESI encoder compared to the frozen encoder in supervised learning approaches also demonstrates the robustness of the learned features for arrhythmia diagnosis.
10

---

Published in Transactions on Machine Learning Research (10/2024)
Evaluation results of arrhythmia diagnosis task under different settings including supervised learn- ing, linear probing (frozen encoder), and fine-tunin ine-tuning. The metric used is the area under the ROC curve (AUC). The best results are highlighted in bold.
Table 2:
Methods
PTB-XL (AUCT)
ICBEB (AUC f)
(Supervised)
LSTM (Strodthoff et al. 2020
0.927
0.964_2.015
XResNet hhooo el al.
2020 0
0.92840.013
0.974
ResNet50 (Strodthoff et al.
2020
0.930
0.969=0.015
Ensemble Strodthoff et al.
2020
0.904
0.975
MLBF-Net [Zhang al. 11
0.931 40.021
MVMSN Yang et al. 2023
0.933
ConvNeXt
Net-NeeX--iyyy WWo et al. 2023
0.918+0.016
0.970=0.012
ConvNeXt-Basn Woo et al.
2023
0.91440.014
0.970=o.012
(Linear Probing)
SimCLR Chen et al.
2020
0.76610.016
0.791=10.02
BYOL Grill et al. 2020
0.77640.014
0.800=0.014
CLOCS Kiyasseh et 2021
0.777 10.012
0.80220.015
LEAVES Yu et al. 2022a
0.792+0.015
0.809
MERL (Liu et al.
2024
0.887
(Ours) ESI-Liny
0.927 +0.099
0.975
(Ours) ESI
0.931 10.008
0.978 0.978_0.006
(Fine-tune)
SimCLR (Chen et al. 2020
0.91610.015
0.968
BYOL Grill et al. 2020
0.925+0.014
0.971=0.014
CLOCS Kiyasseh et al.
2021
0.918+0.013
0.977
CRT ZZhng et al.
2022
0.892
LEAVES Yu et al.
2022al
0.926.1..0p
0.976
(Ours) ESI-tiny
0.935
0.978
(Ours) ESI
0.938 10.011
0.982
4.3.2 Zero-shot Inference
To further assess the learned representations during pretraining, also evaluate the zero-shot learning inference assessment following METS (Li et al. 2023a and MERL (Liu et al. 2024). Other than the zero-shot evaluations, we include the few-shot setting of the existing signal-focused SSL approaches in the comparison, including SimCLR (Chen et al.) 2020). BYOL (Grill et al. 2020). CLOCS (Kiyasseh et al. 2021), and LEAVES Yuu ct al. 2022a), with 3% of the original training set as training samples on PTB-XI. dataset.
Table 3 demonstrates the zero-shot learning inference performance of the proposed ESI method along- side basclines (METS, MERL) and the few-shot finc-tuning results for signal-centered SSL methods. Our proposed method achieves the best performance on both AUC and macro F1 scores compared to all other methods. Additionally, ECG-text pretrained models generally outperformed signal-focused pretraining meth- ods even without samples in fine-tuning. This highlights the improved robustness of representations from multimodal pretraining techniques.
4.4 Downstream Task: ECG-based User Identification
ECG generally shows unique patterns that can distinguish individuals, which makes them suitable for subject
identification tasks and offers potential advantages over other biometric traits (Melzi et al. 2023 For example, compared during usage. In this study, we leverage the PTB-XL (Wagner et al. 2020 and ICBEB (Luu et al. 2018 datasets, also used in the arrhythmia diagnosis task, to design one-shot learning benchmark for ECG
identifying persons with facial images, using ECG can further protect users' privacy
11

---

Published in Transactions on Machine Learning Research (10/2024)
Table 3: Evaluation results of arrhythmia diagnosis task under different settings in zero-shot learning on PTB-XL dataset. The metric used in this table is the area under the ROC curve (AUC) and macro F1 score (Fl-macro). X % represents the percentage of the training set used as the training samples in fine-tuning the pretrained encoder. The best results are highlighted in bold.
Method
AUC +
Fi-macro 1
SimCLR Chen et al.
5% 2020
0.735
0.547
BYOL Grill et al._ 2020
5%
0.752
0.564
CLOCS Kiyasseh et al.
2021
5%
0.765
0.581
LEAVES (Yu et al. 2022a 5%
0.760
0.577
METS Li al. 2023a
0%
0.593
MERL Liu et al
2024 0%
0.757
(Ours) ESI
0%
0.812
0.654
identification. Similar to the arrhythmia diagnosis task in Section 4.3 we focus solely on the test splits of the PTB-XL and ICBEB datasets. For PTB-XL, select 5-second signal sequences from each of the 1907 subjects for both training and testing, which results in 1907-class classification task. Similarly, we select 4-second samples and classes from the ICBEB dataset, which forms a 689-class classification task.
In this task, we compare the performance of the proposed ESI method with various approaches under linear probing and fine-tuning settings. For the specialized supervised methods, we compared our method with established supervised learners, including LSTM, XRResNe1101, ResNet50, and ensemble methods (Strodthoff
ct al.
2020 Besides the supervised lcarning methods, we also evaluated the ESI method against signal-
locused SSL approaches including SimCLR (Chen et al. 2020 BYOL (Grill et al. 2020), CLOCS (Kiyasseh) et al.
2021 and LEAVES (Yu et al. 2022a) under the linear probing and fine-tuning settings. Due to the patient de-identification during pretraining, we cannot perform zero-shot learning for this identification task, as the model has not been exposed to identifiable patient information.
Table summarizes the findings. The results demonstrate that the ESST method significantly outperforms the baselines in both linear probing and fine-tuning settings. Notably, the fine-tuned ESI method achieves substantial accuracy improvement (12.0% and 12.2% on PTB-XL and ICBEB, respectively) compared to the supervised ConyNeXt baselines. Additionally, in linear probing, EST surpasses the signal-focused SSL, methods by a significant margin. These findings highlight the effectiveness of multimodal pretraining with ECG and text data in learning transferable representations for ECG identification.
5 Discussion
In this section, we first discuss the ablations the designed components, including the effectiveness of CQA, the selections of the signal encoder, as well the contributions from captioning loss and contrastive loss to the learned representations. The experiments for ablations are mostly conducted on tiny model variant as ESI-tiny with the ConvNeXt-tiny signal backbone. We also explore the impact of the size of pretraining data as well as the potential misalignment to the proposed method.
5.1 Ablation Study: Component Analysis and Impact
We conduct ablation experiments to assess the contributions of individual components within the pretraining framework. We evaluate the pretrained models' performance both arrhythmia diagnosis and ECG- based user identification tasks using the PTB-XL dataset with the E.S--tiny variant featuring ConyNeXt-tiny signal encoder. The model is evaluated under a linear probing setting with frozen encoders, and the AUC score serves as the primary metric.
The results of this ablation study are summarized Table aa). Removing the Contrastive Question Answer- ing (CQA) module results in a decrease in AUC scores for both tasks. This indicates that CQA contributes to aligning ECG signals with their enriched text annotations. By aligning these modalities, CQA helps the
12

---

Published in Transactions on Machine Learning Research (10/2024)
Table 4: Evaluation results of 1-shot ECG-based user identification task under settings including supervised learning, linear probing, and fine-tuning. The metric used the area under the ROC curve (AUC) and accuracy score (ACC). The best results highlighted in bold.
PTB-XI
ICBEB
Method
AUC f
ACC1 T
AUC
ACC 1
(Supervised)
LSTM (Strodthoff et al.
2020
0.907 10.014
0.444
0.918
0.610
XReeNet101 [Strodthoff et al. 2020
0.915+0.011
0.473-0.016
0.921+0.017
0.623+0.020
ResNet50 (Strodthoft et al.
2020
0.92644.016
0.497=0.021
0.933
0.641
Ensemble Strodthooff et al.
2020
0.930+0.011
0.500=0.015
0.937+0.013
0.653+0.016
ConvNeXt- Tiny Woo et al.
2023
0.918
0.480
0.93610.011
0.647 0.64710.011 10.016
Ne Woo et al.
2023
0.922+0.013
0.494=0.021
0.932+0.013
0.649
(Lineur Probing)
SimCLR Chen al.
2020
0..022
0.185
0.838
0..02
BYOL Grill et al. 2020
0.837
0.240
0.85540.019
0.283
CLOCS Kiyasseh et al. 2021
0.822+0.022
0.207 =0.036
0.841 +0.020
0.235
LEAVES Yu et al. 2022a
0..40
0.248 10.029
0.853 10.022
0.275 0..75 10.029
(Ours) ESI-tiny
0.92340.014
0.510=0.017
0.937+0.012
0..04
(Ours) EST
0.927
0.517
0.94440.010
0.665
Fine-tune)
SimCLR (Chen et al.
2020
0.93640.011
0.522
0.945
0.673
BYOL Grill et al. 2020
0.942+0.011
0.547=0.014
0.951+0.013
0.686+0.012
CLOCS Kiyasseh
2021
0.929
0.516-0.010
0.940
0.66640.000
LEAVES Yu et al. 2022al
0.944
0.550
0.953
0.688 0.68840.00 10.015
(Ours) ESI-tiny
0.96640.010
0.591 0.591=0..14
0.980
0.747 0.747 10.012
(Ours) ESI
0.970+0.009
0.608+0.013
0.985+0.005
0.762
13

---

Published in Transactions on Machine Learning Research (10/2024)
Table 5; Ablation experiments. The evaluation performances on the PTB-XL dataset for both arrhythmia diagnosis and identification tasks. The experiments are under the linear probing setting with frozen encoders, and the evaluation metrics used are the AUC scores. In the following tables, the performances from the most contributing components are highlighted in bold.
(a) Components of the proposed framework
(b) Selection of the signal encoder
w/o
Arrhythmia Identification
Backbone
Params
Arrhythmia
Identification
0.928
0.923
ViT-1D
84.92 M
0.897
0.910
CQA
0.901
0.902
XO11Net001-ID
1.80 M
0.894
0.901
Ccap 0.850
0.913
0.905
ConyNeXt-ting
26.81 M
0.928
0.923
LCon
0.877
ConvNeXt-base
85.56 M
0.932
0.926
Table Ablation exporiment on pretraining ESI-Tiny on the MIMIC-IV-EG data only. The table compares the performance of the model pre-trained exclusively on MIMIC-IV-EGG with the model pre-trained on the full dataset across different tasks and datasets.
Pretraining Arrhythmia (Lincar Probing) Set
Arrhythmia (Zero-shot)
Identification (Lincar Probing)
PTB-XL
ICBEB
PTB-XL
ICBEB
PTB-XL
ICBEB
MIMIC
0.912
0.967
0.792
0.832
0.917
0.934
All
0.928
0.976
0.807
0.842
0.923
0.937
diff'
4 0.016
40.009
10.015
+ 0.010
4. 0.006
+ 0.003
model learn more robust representations that capture the inherent relationships between ECG patterns and associated diagnoses. Ablating the captioning loss Ccap also leads to a performance decline on both arrhyth- mia diagnosis and ECG identification, which suggests that the model benefits from explicitly generating captions during pretraining. The most substantial impact performance is observed when removing the contrastive loss LC..- By contrasting ECG with different textual representations, the model is encouraged to identify subtle variations that are relevant to downstream tasks.
5.2 Ablation Study: Impact of Pre- Pre-Training with MIMIC-IV-ECG Only
As the MIMIC-IV-ECG data is the single dataset that contains the most ECG samples used in pretraining our ESI models, to investigate the impact of using only the MIMIC-IV-ECG dataset for pre-training, we conducted an experiment where the model was pre-trained exclusively on MIMIC-IV-ECG. We then eval- nated the model's performance on the PTB-XL and ICBEB datasets, focusing on three tasks: arrhythmia detection (both linear probing and zero-shot) and subject identification (linear probing).
Table 0 presents the performance results across these tasks. The results indicate that pre-training on the full dataset yields better performance across all tasks and datasets compared to pre-training exclusively on MIMIC-IV-EGG Specifically, there is noticeable improvement in both arrhythmia detection and subject identification tasks when additional datasets are included in the pre-training process. For the arrhythmia detection task, the difference in performance is more substantial in the zero-shot setting, where the model pre-trained on all datasets achieves an improvement of 0.015 in PTB-XL and 0.014 in ICBEB. In the identification task, the model pre-trained on all datasets only shows a slight improvement.
These results suggest. that while MIMIC-IV-ECG provides strong foundation for pre-training, and the inclusion of additional datasets can further enhance the model's ability to generalize across different tasks and datasets. Also, for the zero-shot learning tasks, learning from the samples and annotating information from the same dataset during pretraining, e.g., PTB-XL, could help the downstream tasks accordingly.
5.3 Ablation Study: Selection of Backbone for Signal Encoder
We investigate the impact of various backbone architectures for the signal encoder on the model's perfor- mance. Prior research vision-text. pretraining informed our selection of candidate backbones, including
14

---

Published in Transactions on Machine Learning Research (10/2024)
PTB-XL
1.0
PTB-XL
0.95
ICBEB
ICBEB
Performance (AUC)
0.8
0.90
MMD
0.6
0.85
0.80
0.4
0.75
0.2
0.70
0
50
100
200 300
400 500
660
50 100
200
Pretraining size (K)
Pretraining size (K)
Figure 5: Performances of linear probing infer- ence in arrhythmia diagnosis (AUC) PTB-XL and ICBEB data using the pretrained encoders with varying training samples.
Figure 6: Distribution difference mcasured by maximum mean discrepancy (MMD) between the pretraining set and test set using encoders with varying training samples.
VIT (Dosovitskiy et al. 2020) and ConvNeXt (Woo et. al. 2023) architectures. Additionally, considering the strong performance of 1D1011 in supervised arrhythmia diagnosis (Strodthoff et al., 2020) as shown in Table 2 we examine its potential as foundation encoder after multimodal pretraining.
Table 5 b) summarizes the results. ConvNeXI-basee models (ConvNeXl-Liny and ConvNeXX-basse) achieved the highest AUC scores on both arrhythmia diagnosis and ECG identification tasks. Notably, ConvNeXt- base, the largest model with the most parameters (85.56M), provides the best overall performance (AUC of 0.932 for arrhythmia diagnosis and 0.926 for ECG identification). ConvNeXt-tiny, a more parameter-offciem option (26.81M), shows a slightly lower performance compared to the base version, but also achieves petitive AUC scores with significantly lower parameter footprint. With the same-level size Conv models in both tasks, the ViT model shows substantially lower performances, which might indicate the con- volutional kernel could be more suitable for processing ECG signals. Moreover, the XResNettD101 backbone displays lower AUC scores compared to ConvNeXt architectures. This can be due to the significantly smaller parameter size of the Xets.Net model.
5.4 Ablation Study: Impact of Pretraining Data Sizes and Distribution Shifts
In this ablation study, we also investigale the impact pretraining dataset size on the performance of our ESS-tiny model in two arrhythmia classification datasets, including PTR-XT, and ICBEB. We artificially change the size of the unlabeled pretraining dataset from all available samples by randomly sampling and selecting from all the pretraining samples. After ESI-tiny encoders trained, the models are then evaluated on both datasets using linear probing approach with frozen encoder weights, and the AUC score is employed as the performance metric.
The results presented Figure 5 demonstrate a consistent performance improvement as the pretraining dataset size increases for both PTB-XI. and ICBEB datasets. The model's performance improves most substantially the carly stages of incrcasing the protraining dataset size. This indicates that a modest amount of pretraining data can help substantial gains in learning informative representations.
Additionally, to understand the effect of pretraining on distribution shift, measure the Maximum Mean Discrepancy (MMD) between the pretraining dataset and cach test set (PTB-XL and ICBEB) as the pre- training size increases. As shown in Figure 6 the distribution shift between the pretraining and test sets decreases as the pretraining dataset size increases. This suggests that larger pretraining datasets help the model learn representations that are more generalizable to the target datascts. The decrease in MMD corre- lates with the improvements in AUC scores observed in Figure 5 which potentially indicates that reducing distribution shift through larger pretraining sets contributes to better performance the downstream tasks.
15

---

Published in Transactions on Machine Learning Research (10/2024)
0.90
PTB-XL
PTB-XL (Random Init)
0.85
Performance (AUC)
ICBEB
ICBEB (Random Init)
0.80
0.75
0.70
0.65
0
510
20
40
60
80
100
Misalignment (%)
Performances of linear probing inference in arrhythmia diagnosis (AUC) on PTB-XL and ICBEB data using pretrained encoders with varying ECG-text misalignment ratios. Models trained with randomly initialized weights are included as baselines.
Figure 7:
5.5 Ablation Study: Impact of ECG-text Misalignment
To investigate the impact of potential misalignments between ECG signals and their corresponding text descriptions on the proposed ESI method, we conduct an experiment on ESI-tiny with a sub-training set of 100K ECG-text pairs. We introduce various degrees of misalignment by randomly shuffling a percentage of E.G-text pairs in our pretraining dataset. The model was then evaluated on two arrhythmia classification datasets with linear probing approach with frozen encoder weights.
Figure 7 shows the results of the experiment. As the percentage of misaligned pairs increases, the perfor- mance of ESI-tiny decreases substantially on both datasets. This indicates that the aligmment between the pretraining sample pairs plays an essential role in lcarning robust representations. Notably, increasing mis- alignment can lead to performance that is worse than using un-pretrained models with random initialization, which also highlights the critical importance of accurate ECG-text pairing in pretraining.
6 Conclusion
This study introduces a novel multimodal contrastive pretraining framework to enhance the quality and robustness of representations learned from ECG signals. To address the lack of descriptive text associated with ECGs, we propose vetrieval-uugmented generation (RAG) pipeline called the Cardio Query Assistant (CQA). This pipcline generates detailed textual descriptions for ECG data with demographic information, potential conditions, and waveform patterns. Inspired by the success of multimodal pretraining strategies in vision-language tasks, we develop the ECG Semantic Integrator (FIST). This framework integrates both contrastive and captioning capabilities to foster a deeper semantic understanding of ECG signals. Our evaluation validates the effectiveness the proposed approach across several downstream tasks. The ESI method demonstrates improvement in arrhythmia diagnosis and F.GGlaased user identification tasks by outperforming strong basclines that cover superviscd learning and SSL approaches. These results with the ablation studies highlight the benefits of multimodal learning for ECG analysis and the value of integrating captioning loss with contrastive pretraining, Beyond ECG, we believe the proposed CQA and ESI frameworks hold the potential for applications to other types of biomedical time series data, where contextual information can be leveraged to enhance the representation learning and downstream analysis.
On the other hand, this study limited by the use of 10-second ECG signals only pretraining. While our approach demonstrates effectiveness on this data, real-world ECG recordings can vary significantly length and may contain more diverse features. In future work, we plan investigate the impact of using more diverse ECG signals the performance of the proposed framework.
16

---

Published in Transactions on Machine Learning Research (10/2024)
Acknowledgments
This work was supported by National Science Foundation (# 2047296) and National Institute Health (# R01DA059925)
References
Clas Baran Baloglu, Muhammed Talo, Ozal Yildirim, Ru San Tan, and Rajendra Acharya. Classification of myocardial infarction with multi-lead oog signals and deep cnn. Pottern Recognition Letters, 122:23 30, 2019.
Behnam Behinaein, Anubhav Bhatti, Dirk Rodenburg, Paul Hungler, and Ali Etemad. A transformer architecture for stress detection from ecg. In 2021 International Symposium on Wearable Computers, pp. 132-134, 2021.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for rastivve learning of visual representations. In International conference on machine learning, pp. 1597-169000 1607. PMLR, 2020.
Jacob Devlin, Ming-Wei Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidircctional transformers for language understanding. orXiv preprinl ar urXiv:1810.0 04800, 2018.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Un- terthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, al. An image is worth 16x16 words: Transformers for image recognition scale. arXiv preprint arXiv:2010. 11929, 2020.
Emadeldeen Eldele, Mohamed Ragab, Zhenghua Chen, Min Wi, Chee Keong Kwoh, Xiaoli Li, and Cun- tai Guan. Time-series representation learning via temporal and contextual contrasting. arXiv preprint arXiv:2106.144 14112 2021.
Oliver Faust, U Rajendra Acharya, EYK Ng, and Hamido Fujita. A review of ecg-based diagnosis support systems for obstructive sleep apnea. Journal of Mechanics in Medicine and Biology, 16(01) ): 1444004 2016.
Tomas B Garcia. 12-lead ECG: The art of interprelation. Jones Bartlett Learning, 2015.
Timnit Gebru, Jamic Morgenstern, Briana Vecchione, Jennifer Wortman Vaughan, Hanna Wallach, Hal Daumé Tii, and Kate Crawford. Datasheets for datasets. Communiccttioss of the ACM, 64(12): 86 86-92, 2021.
Bryan Gopal, Ryan Han, Gautham Raghupathi, Andrew Ng, Geoff Tison, and Pranav Rajpurkar. 3kg: Con- trastive learning 12-lcad clectrocardoorrans using physiologically-issirre augmentations. Machine Learning for Health, pp. 156-167. PMLR, 2021.
Brian Gow, Tom Pollard, Larry A Nathanson, Alistair Johnson, Benjamin Moody, Chrystinne Fernandes, Nathaniel Greenbaum, Seth Berkowitz, Dana Moukheiber, Parastou Eslami, et al. seaii-iv-ecg-diannostt electrocardiogram matched subset. Type: dataset, 2023.
Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Flena. Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent-a new approach to self-supervised learning. Advances in neural information processing syslems, 33:21271 11221271-21284, 2020.
Jian Guan, Wenbo Wang, Pengming Feng, Xinxin Wang, and Wenwu Wang. Tow-dimensionall denoising embedding transformer for eeg classification. In ICASSP 2021-2021 IEFE International Conference Acoustics, Speech and Signal Processing (ICASSP), pp. 1285 1289. IEEE, 2021.
TH Haapaniemi, Ville Pursiainen, JT Korpelainen, HV Huikuri, KA Sotaniemi, and VV Myllyl. Ambulatory ecg and analysis of heart rate variability in parkinson's discase. Journal of neurology, neurosurgery & psychiatry, 30(3):305-311 2001.
17

---

Published in Transactions on Machine Learning Research (10/2024)
Awni Y Hannun, Pranav Rajpurkar, Masoumch Haghpanahi, Geoffrey H Tison, Codie Bourn, Mintu P Turakhia, and Andrew Y Ng. Cardiologist-leeel arrhythmia detection and classification in ambulatory electrocardiograms using a deep neural network. Nature medicine, 25(1):65 69, 2019.
Jane Huff. ECG workout: Exercises in arrhythmia interpretation. Lippincott Williams & Wilkinn, 2006.
Rahul Jain, Robin Singh, Sundermurthy Yamini, and Mithilesh Das. Fragmented ecg as risk marker in cardiovascular diseases. Current Cardiology Revieus, 10(3):277-286 2014.
Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana. Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. In International conference on machine learning, pp. 4904-4916. PMLR, 2021.
Enbiao Jing, Haiyang Zhang, ZhiGang Li, Yazhi Liu, Zhanlin Ji, and Ivan Ganchev. Ecg heartbeat classi- fication based on an improved resnet-18 model. Computational and Mathematical Methods in Medicine, 2021, 2021.
Dani Kiyasseh, Tingting Zhu, and David A Clifton. Clocs: Contrastive learning of cardiac signals across space, time, and patients. In International Conference on Machine Learning, pp. 1606-5615 PMLR, 2021.
Paul Kligfield, Leonard S Gettes, James Bailey, Rory Childers, Barbara J Deal, E William Hancock, Gerard Van Herpen, Jan Kors, Peter Macfarlane, David M Mirvis, al. Recommendations for the standard- ization and interpretation of the electrocardiogram: part i: the electrocardiogram and its technology: a scientific statement from the american heart association electrocardiograply and arrhythmias commitlee, council clinical cardiology; the american college of cardiology foundation; and the heart rhythm society endorsed by the international society for computerized electrocardioloy. Circulation, 130010):100--132, 2007.
Sravan Kumar Lalam, Hari Krishna Kunderu, Shayan Ghosh, Harish Kumar, Samir Awasthi, Ashim Prasad, Francisco Lopez-Jimenez, Zachi I Attia, Samuel Asirvatham, Paul Friedman, et al. Ecg representation learning with multi-modal ehr data. Transactions on Machine Learning Research, 2023.
Jun Li, Che Liu, Sibo Cheng, Rossella Arcucci, and Shenda Hong. Frozen language model helps ecg zero-shot learning. arXiv preprint arXin:2303.122 12311, 1, 2023a.
Jun Li, Che Liu, Sibo Cheng, Rossella Areucci, and Shenda Hong. Frozen language model helps eog zero-shot learning. In Medical Imaging with Deep Learning, pp. 402 415. PMLR, 2024.
Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping languago-imas image pre-training for unified vision-language understanding and generation. In International conference on machine learning, pp. 12888 12900. PMLR, 2022.
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In International conference machine learning, pp. 19730-19745 PMLR, 2023b.
Che Liu, Sibo Cheng, Miaojing Shi, Anand Shah, Wenjia Bai, and Rossella Arcucci. Imitate: Clinical prior guided hierarchical vision-language pre-training. arXiv preprint arXiv:2310. 0.3055,775555 2023.
Che Liu, Zhongwei Wan, Cheng Ouyang, Anand Shah, Wenjia Bai, and Rossella Arcucci. Zero-shot eog classification with multimodal learning and time clinical knowledge enhancement. arXie preprint NNiu:2403.06652 2024.
Feifei Liu, Chengyu Liu, Lina Zhao, Xiangyu Zhang, Xiaoling Wu, Xiaoyan Xu, Yulin Liu, Caiyun Ma Shoushui Wei, Zhiqiang He, et al. An open access database for evaluating the algorithms of electrocardio- gram thy and morphology abnormality detection. Journal of Medical Imaging and Health Informatics, 8(7):1368-138) 1373, 2018
Xinwen Liu, Huan Wang, Zongjin Li, and Lang Qin. Deep learning in ecg diagnosis: A review. Knowledge- Based Systems, 227:107187, 2021.
18

---

Published in Transactions on Machine Learning Research (10/2024)
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arxiv preprint Xr:: 16692, 2019.
Temesgen Mehari and Nils Stff Self-supervised representation learning from 12-lead eog data. Com- puters in biology and medicine, 141:105114, 2022.
Pietro Melzi, Ruben Tolosana, and Ruben Vera-Rodriguez Eeg biometric recognition: Review, system proposal, and benchmark evaluation. IEEE Access, 2023.
Mendable. Langchain: A framework for developing applications powered by language models. Software framework, 2023. Retrieved from htttp:///gtthhoo...cmmmmaaab//a/
Lingxiao Meng, Wenjun Tan, Jiangang Ma, Ruofei Wang, Xiaoxia Yin, and Yanchun Zhang. Enhancing dy- namic eog heartbeat classification with lightweight transformer model. Artificial Intelligence in Medicine, 124:102236, 2022.
Ana Mincholé and Blanca Rodriguez. Artificial intelligence for the clectrocardiogram. Nature medicine, 25 (1):22-23, 2019.
Sheikh Shanawaz Mostafa, Fábio Mendonça, Antonio G. Ravelo-Garcia, and Fernando Morgado-Dias. A systematic review of detecting sleep apnea using deep learning. Sensors, 19(22):4934, 2019.
Annamalai Natarajan, Yalc Chang, Sara Mariani, Asif Rahman, Gregory Boverman, Shruti Vij. and Jonathan Rubin. A wide and deep transformer neural network for 12-lead eog classification. In 2020 Computing in Cardiology, pp. 1-4. IEEE, 2020.
Jungwoo Oh, Hyunseung Chung, Joon- loon-myoung Kwon, Dong-gyun Hong, and Edward Choi. Lead-agnostic self-supervised learning for local and global representations of electrocardiogram. In Conference Health, Inference, and Learning, pp. 338 353. PMLR, 2022.
Aaron den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint arXim:1807.097488 2018.
OpenAl. Embedding- openai. Software AP1, 2023. Retrieved from http://ppaattrr.opppaaa.ooo/o//p guides//ebedddnng
Boris Pyakillya, Natasha Kazachenko, and Nikolay Mikhailovsky. Deep learning for eog classification. In Journal of physics: conference series, volume 913, pp. 012004. IOP Publishing, 2017.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748 8763. PMLR, 2021.
Giovanna Sannino and Giuseppe De Pietro. A deep learning approach for ecgebased heartbeat classification for arrhythmia detection. Future Generation Computer Systems, 86:446-455, 2018.
Pritam Sarkar and Ali Etemad. Self-supervised eeg representation learning for emotion recognition. 1EEE Transactions on Affective Computing, 13(3):1541-1554, 2020.
Sandra Smigiel, Krzysztof Palczyúski, and Damian Ledziński. Eeg signal classification using deep learning techniques based on the ptb-xl dataset. Entropy, 2391:1122 2021.
Nils Strodthoff, Patrick Wagner, Tobias Schaeffter, and Wojciech Samek. Deep learning for eeg analysis: Benchmarks and insights from ptb-xl. JEEE journal of biomedical and health informatics, 151))115 2020.
Oriol Vinyals, Alexander Toshev, Samy Bengio, and Dumitru Erhan. Show and tell: A neural image caption generator. In Proceedings of the IEEE conference on computer vision and pattern recognétion, pp. 3156- 3164, 2015.
19

---

Published in Transactions on Machine Learning Research (10/2024)
Patrick Wagner, Nils Strodthoff, Ralf-Dieter Bousseljot, Dieter Kreiseler, Fatima I Lunze, Wojeiech Samek, and Tobias Schaeffter. Ptb-xl, a large publicly available electrocardiograply dataset. Scienlific data, 7(1): 154, 2020.
Zhongwei Wan, Che Liu, Mi Zhang, Jie Fu, Benyou Wang, Sibo Cheng, Lei Ma, César Quilodrán- and Rossella Arcucci. Med-unic: Unifying cross-lingual medical vision-language pro-training by diminishing bias. Advances in Neural Information Processing Systems, 36, 2024.
Peng Wang, An Yang, Rui Men, Junyang Lin, Shuai Bai, Zhikang Li, Jianxin Ma, Chang Zhou, Jingren Zhou, and Hongxia Yang. Ofa: Unifying architectures, tasks, and modalities through a simple sequence-to- sequence learning framework. In International Conference on Machine Learning, pp. 23318 23340. PMLR, 2022.
Zirui Wang, Jiahui Yu, Adams Wei Yu, Zihang Dai, Yulia Tsvetkov, and Yuan Cao. Simvlm: Simple visual language model pretraining with weak supervision. arXiv preprint orXiv:2108. 10904, 2021.
Hao Wen and Jingsu Kang. torch eegh An ECG Deep Learning Framework Implemented using PyTorch, 2022. URL http:::t//eeeee
org/record/6435048
Sanghyun Woo, Shoubhik Debnath, Ronghang Hu, Xinlei Chen, Zhuang Läu, In So Kweon, and Saining Xie. Convnext v2: Co-designing and scaling convnets with masked autoencoders. In Proceedings of the IEEE/CVP Conference on Computer Vision and Pollern Recognition, pp. 16133 16142, 2023.
Genshen Yan, Shen Liang, Yanchun Zhang, and Fan Liu. Fusing transformer model with temporal features for eog heartbeat classification. In 2019 IEEE International Conference on Bioinformatics and Biomedicine (BIBM), pp. 898-905. 905. TEFF, 2019.
Shunxiang Yang, Cheng Lian, Zhigang Zeng, Bingrong Xu, Junbin Zang, and Zhidong Zhang. A multi-view multi-scale neural network for multi-label cog classification. IEEE Transactions on Emerging Topics in Compulational Intelligence, 2023.
Michihiro Yasunaga, Jure Leskovec, and Percy Liang, Linkbert: Pretraining language models with document links. arxiv preprint arXiv:2203. 15827, 2022.
Kihyun You, Jawook Gu, Jiyeon Ham, Beomhee Park, Jiho Kim, Eun K Hong, Woonhyuk Back, and Byungseok Roh. Cxr-clip: Toward large scale chest x-ray language-image are-training. In International Conference on Medical Image Computing and Assiste Intervention, pp. 101-111. Springer, 2023.
Han Yu, Huiyuan Yang, and Akane Sano. Leaves: Learning views for time-series data in contrastive learning. arXiv preprint arXim:2210.073.4 2022a.
Han Yu, Huiyuan Yang, and Akane Sano. Eeg-sl: Electrocardiogram (eeg) segment learning, deep learning method for ecg signal. ar preprint arXiv:2310.00m 00818, 2023.
Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung, Mojtaba Seyedhosseini, and Yonghui Wu. Coca: Contrastive captioners are imago-text foundation models. arXio preprint arXiv:2205.c 009917, 2022b.
Lu Yuan, Dongdong Chen, Yi-Ling Chen, Noel Codella, Xiyang Dai, Jianfeng Gao, Houdong Hu, Xuedong Huang, Boxin Li, Chunyuan Li, et al. Florence: A new foundation model for computer vision. arXiv preprinl ar Xiv::221 1. 11111.11400 11432, 2021.
Xiaohua Zhai, Xiao Wang, Basil Mustafa, Andreas Steiner, Daniel Keysers, Alexander Kolesnikov, and Lucas Beyer. Lit: Zero-shot transfer with locked- image text tuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 18123-18133, 2022.
Jing Zhang, Dong Liang, Aiping Liu, Min Gao, Xiang Chen, Xu Zhang, and Xun Chen. Mibf-net: A multi- lead-branch fusion network for multi-class arrhythmia classification using 12-lead eeg. IEEE journal of translational engineering in health and medicine, 9:1-11, 2021.
20

---

Published in Transactions on Machine Learning Research (10/2024)
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi Bertscore: Evaluating text generation with bert. In International Conference on Learning Represenlations, 2020. URL https:
Wenrui Zhang, Ling Yang, Shijia Geng, and Shenda Hong. Self-supervised time series representation learning via cross reconstruction transformer. arXiv preprint arXiv:2205. 09928,, 2022.
Jianwei Zheng, Huimin Chu, Daniele Struppa, Jianming Zhang, Sir Magdi Yacoub, Hesham El- El-Askary, Anthony Chang, Louis Ehwarhemuopha, Islam Abudayych, Alexander Barrett, et al. Optimal multi-stage arrhythmia classification approach. Scientific reports, 10(1):2898) 2020.
Zhaowei Zhu, Han Wang, Tingling Zhao, Yangming Guo, Zhuoyang Xu, Zhuo Liu, Siqi Liu, Xiang Lan, Xingzhi Sun, and Mengling Feng. Classification cardiac abnormalities from eog signals using se-resnet. In 2020 Computing in Cardiology, pp. 1-4. IEEE, 2020.
21

---

Published in Transactions on Machine Learning Research (10/2024)
A Ablation Study on CQA
The introduction of Retrieval-Angmennte Generation (RAG) in CQA in this study plays a crucial role in enriching the pretraining dataset by generating detailed ECG waveform descriptions. In this ablation study, we investigate the impact of different RAG settings on performance.
To evaluate the performance of our RAG-generated descriptions, we utilize the human-annotated ECG waveform descriptions from (Wen & Kang 2022 as references. This dataset includes annotations for 28 types of arrhythmia. We generate corresponding descriptions using our RAG method and compared them against the reference descriptions using BERTScore (Zhang et al. 2020) with the RoBERTTa-laree model {Liu ct al. 2019 BERTSccore is metric designed to evaluate the similarity between generated text and reference text by leveraging pre-trained contextual embeddings from transformer models. BERTScore computes the similarity of embeddings to provide a more nuanced assessment of semantic alignment, and there are three main metrics served in BERTScore:
Precision measures how well the tokens in the generated text match with the tokens in the reference text, which indicates how accurately the generated content reflects the reference.
Recall assesses how well the tokens in the reference text are captured by the generated text, which reflects the completeness of the generated content.
F1 Score is the harmonic mean of Precision and Recall, which provides balanced measure of both accuracy and completeness in the semantic alignment between the generated and reference texts.
A.1 Impact of Top-k in Vector Search
In this section, we examine the effect of varying the top-k parameter in the vector search interface the model's performance. The top-k parameter controls how many of the top-ranked candidates are considered in the generation process. We evaluated the model's performance by measuring Precision, Recall, and F1 score generated by BERTScore for different values of top-k. The mean and standard deviation of these metrics across multiple runs are summarized in Table
Table 7: BERTScores across different Top-k values
Top K
B Precision f
B Recalt f
B B11score
1
0.8190 (0.0128)
0.8326 (0.0237)
0.8256 (0.0147)
2
0.8197 (0.0159)
0.8402 (0.0156)
0.8296 (0.0090)
3
0.8169 (0.0160)
0.8433 (0.0155)
0.8297 (0.0106)
4
0.8159 (0.0157)
0.8447 (0.0160)
0.8299 (0.0111)
5
0.8153 (0.0172)
0.8447 (0.0167) 0.8295 (0.0110)
The results indicate that as the top-k value increases, the Recall generally improves slightly, which suggests that considering more candidates helps in retrieving more relevant information. However, the mean Precision tends to decrease marginally, which reflects a trade-off where including more candidates might introduce less relevant information. The F1 score, which balances both Precision and Recall, remains relatively stable across different top-k values. This indicates that the model maintains a consistent performance overall.
Our findings suggest that varying the top-k parameter has a limited but consistent impact on RAG perfor- mance. The choice of top-k can be adjusted based on the specific needs of the task, with higher values favoring recall at a slight cost to precision, while lower values provide a more precise but slightly less comprehensive output. In this study, set the top-k value as 2.
A.2 Impact of the Number of Textbooks in RAG
We investigate the impact of the number of textbooks used as the knowledge base in the CQA pipeline. The purpose of this ablation study is to assess whether the inclusion of more fewer textbooks affects the
22

---

Published in Transactions on Machine Learning Research (10/2024)
Table 8: BERTScores across different numbers of textbooks used RAG
Textbooks
B Precision f
B Recall 1
B Seore
No Textbook
0.7617 (0.0187)
0.8154 (0.0142)
0.7928 (0.0126)
Garcia (2015)
0.8169 (0.0142)
0.8290 (0.0175)
0.8217 (0.0111)
Huft
2006
0.8182 (0.0145)
0.8395 (0.0170)
0.8285 (0.0100)
Both
0.8197 (0.0159)
0.8402 (0.0156)
0.8296 (0.0090)
quality of the generated ECG waveform descriptions. The textbooks used in this study 'ECG Workout: Erercises In Arrbythmia Interpretation' " by Huffi (2006) and '12-Lead ECG; The Art of Interpretation by Garcia (2015). We conducted experiments using three different configurations: using no textbooks, using each textbook individually, and using both textbooks combined. The results are summarized in Table S
The results show a clear trend that incorporating textbooks into the RAG pipeline improves the perfor- mance across all BERTScore metrics (Precision, Recall, and F1 Score). Specifically, using both textbooks in combination yields the highest scores, with Precision of 0.8197, a Recall of 0.8402, and F1 score of 0.8296. This suggests that the inclusion of multiple sources of knowledge enrichos the generated descriptions. When using only one textbook, Hull (2006) slightly outperforms Garcia (2015) in terms of all three metrics, although the difference is incremental. This indicates that while each textbook independently contributes to the generation quality, combining them provides a more robust knowledge base for the RAG process. contrast, the absence of textbooks results in a significant drop in performance, with the lowest scores across all metrics. This highlights the importance of incorporating domain-specific knowledge to improve the rel- evance and accuracy of LLMs in generating content. Thus, in this study, using multiple textbooks in the RAG pipeline not only increases the quality of the generated ECG descriptions but also ensures consistency in the output.
A.3 Impact of Different LLM Prompting Strategies
In this subsection, we investigate the effect of varying the prompt structure used in the LLM within the RAG pipcline on the quality of the generated ECG waveform descriptions. To assess consistency of the generation across different prompts, we conducted an ablation study by rephrasing our slandard prompt, "How is (ECG condition) / reflected in 12-lead ECG?', into five different variations such as "Describe how (ECG condition, appears in a 12-lead ECG. and "What are the ECG findings associated with (ECG condition) in a 12-lead setup?' The alternative prompts used are designed to explore whether slight changes in wording would affect the performance of the RAG pipeline. Despite these variations, the underlying query remains focused on generating descriptions of how specific ECG conditions are in a 12-lead ECG.
The results demonstrate remarkable consistency across all rephrased prompting strategies, with an average mean F1 score of 0.8295 and very low standard deviation of 0.0003 across the variations of prompts. This indicates that the CQA's s ability to generate relevant descriptions is robust to variations in the phrasing of the prompt. This robustness allows for some flexibility in how prompis are formulated without hurting the quality of the generated content.
B Datasheet for ECG Datasets Used for Training ECG Model
In this section, we compile a dataset for the used training data following Gebru et al. 20211)
B.1 Motivation
The datasets were collected and processed to facilitate the development and evaluation of auto- mated ECG processing algorithms, particularly for training a foundation model that extracts generalized embedding from ECG signals. These datascts provide large number of ECG recordings paired with human or machine annotations and metadata, which enables robust training of machine learning models in the medical domain.
Purpose:
23

---

Published in Transactions on Machine Learning Research (10/2024)
The datasets were created by institutions including PhysioNet, Chapman University, Shaoxing People's Hospital, and Plysikalisch-Techhiedd Bundesanstalt (PTB).
Creators:
Development and release of these datasets were supported by various grants, including the NIH, the Kay Family Foundation, and the Berlin Big Data Center.
Funding:
B.2 Composition
Instances:
MIMIC-IV-EG:
Contains diagnostic 12-lead ECG recordings, with each instance representing an ECG signal paired with machine srcchine-geneatee annotations and metadata.
PTB-XL: Comprises 17,415 clinical 12-lead ECG records, annotated by cardiologists, to cover a wide range of diagnostic categories.
Chapman-Shaoxing: Includes 45,152 12-lead ECGs from patients, annotated by professional ex- perts, with conditions such as atrial fibrillation, myocardial infarction, and other arrhythmias.
Volume:
MIMIC-IV-ECG: The dataset includes thousands of ECG records.
PTB-XL: Contains 17,415 ECG records.
Chapman-Shaoxing: Contains 45,152 ECG records.
Data Features: Each dataset includes ECG waveform data, demographic information (age, sex, etc.), and clinical annotations related cardiac conditions. PTB-XT and Chapman-Shaoxing datasets also include metadata on the diagnostic process, such as SCP-ECG statements.
B.3
Collection Process
Data Acquisition:
MIMIC-IV-ECG:
Data was collected as part of routine clinical care and stored in the MIMIC-IV
database.
PTB-XL: ECG signals were collected using devices from Schiller AG over period of seven years, with annotations provided by cardiologists.
Chapman-Shaoxing: Data was collected from multiple hospitals, with ECG signals recorded and labeled by licensed physicians.
Ethical Considerations: All datasets publicly available and de-identified to protect patient privacy.
The collection of the original data was approved by relevant institutional ethics committees. No new data was collected in this study.
B.4 Preprocessing/Claannng/LLLe
MIMIC-IV-ECG; Machine-generated annotations were used for labeling. The data cleaned to remove noise and artifacts.
PTB-XL: FCG data was preprocessed to ensure consistency in the waveform format, and SCP-ECG codes were applied for standardized labeling.
Chapman-Shaoxing ECG data was denoised, and diagnostic information was verified by multiple physi- cians before being stored in structured format.
24

---

Published in Transactions on Machine Learning Research (10/2024)
B.5 Uses
Current Uses: These dalasets have been used for developing and evaluating machine learning models for ECG analysis, particularly in pretraining ECG models and detecting arrhythmias.
Potential Uses: The datascts can be used for broader research cardiac health, including developing diagnostic tools and studying the epidemiology heart conditions.
Restrictions: While the datasets de-identified, users should still adhere ethical guidelines when using the data for rescarch.
B.6 Distribution
The datasets are publicly available through the PhysioNet platform under the Creative Commons Attribution 4.0 International License.
Access:
Regulatory Issues:
There are specific regulatory restrictions, but users must follow the licensing terms.
B.7
Maintenance
Support:
Users contact the dataset maintainers through the PhysioNet platform for support ques- tions.
Updates: Any updates or new versions of the datasets will be announced on the PhysioNet platform.
Community Contributions: Researchers are encouraged to contribute their findings and improvements back the community through publications or direct contributions to the dataset repository.
25


Reviews:
[{"review_id": "qjqNb9Wr9t", "reviewer": "Reviewer_8LVf", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n\n1. This paper is clearly written and explained and tackles an important problem in the medical domain to build multimodal pretrained representations for ECG diagnosis.\n2. It is well situated in the literature both in the previous relevant multimodal models and the ECG-specific models that are currently state of the art. (note: I’ve pointed out some other works to be cited as well as newer methods to look into).\n3. The experimental methodology is clearly explained and experimentally sound.\n4. The evaluations compose of zero shot evaluations, linear probing and evaluations of fine-tuned components to downstream tasks which allow us to both look at model representations as well as the ability to generalise which is good to see.\n5. This method shows improved performance over the previous best models (even when fully supervised) and that the fine-tuned ESI model outperforms all previous models when looking at AUC by a small margin.\n6. They also show the robustness of the learned representations by evaluating that this generalises to downstream tasks better than previous models that learn multimodal representations.\n7. Question: What is the demographic information in the dataset? Apologies if I missed this but it’s not present clearly in the section that describes the data and is an important component of it. \n8. Question: What is the reasoning behind using a ConvNet encoder vs. another encoder (ResNet, or more complicated prefix-lm type encoders). The reasoning behind the encoder choices would be good to explain in the paper for future work to build off of/compare other encoders to if needed.\n9. Question: Have the authors experimented with augmenting the generated data with perturbations, entailments etc., to expand the dataset?\n10. Question: Have the authors evaluated the benefit of a RAG-based approach to other sampling methods for the dataset to create a pretraining dataset? It’s great to see good performance with RAG, but it would be good to verify that this is superior to other ways of retrieving from existing large resources to create a new dataset that you then evaluate by looking at model performance on this data. \n11. Question: When using BioLinkBert for the text encoding it would be good to have a few sentences outlining how this is done and what is different from a standard BERT encoder (or a baseline that is not specific to biomedical data).\n\nWeaknesses:\n\n1. The introduced model/architecture is largely based on the CLIP model (Radford et. al., 2021) from a few years ago. While this is still a very competitive model and these representations do form the basis of the other stronger models that are currently SOTA, there have been other approaches (e.g., PrefixLM, VITs, fusion with cross attention https://arxiv.org/abs/2108.10904, https://arxiv.org/abs/2204.14198, https://arxiv.org/abs/2102.10407) in the last few years that have far surpassed this in multimodal pre-training. \n2. The data generation and RAG-retrieval process is described  clearly but attributes of the data itself are not very clear. If the authors could add a table or figure that outlines the components of the data (waveform, demographics) with an example snippet from the dataset that would illustrate this very nicely. This coupled with a datasheet that gives us summary statistics of demographics, waveforms, text, any private information etc., would be good to ensure any safety concerns about the dataset as well.\n3. A datasheet that outlines the data generated would be good to have. This would be particularly helpful for this dataset since (a) it could potentially concern private human information and (b) the exact form of the data might be unfamiliar to researchers given the various modalities, it would be nice to document all of this in one place for clarity. See this paper and example datasheets here. https://arxiv.org/abs/1803.09010\n4. The metrics reported are largely just AUC for the three different categories (supervised, probing, fine-tuning) and we see a small difference of around 0.01 between the best performing models. To properly evaluate whether this effect is significant, it would be good to have accuracy measures with confidence scores, averages over at least 5 different samples of the model for every sample, and calibration scores of the model as well if possible.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces a contrastive pretraining approach for multimodal data (ECG signals and text) as well as a RAG-based approach to create diverse training datasets that serve as pre-training data for the contrastive approach. They show that when using their method to create a pretraining dataset that is then used with their method + objective function for training, this can learn generalisable representations for ECG, as evaluated on several downstream tasks (e.g., arrhythmia detection) in the medical domain. They also show that this improves over baselines that have more supervision—both supervised and self-supervised methods.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. Add an overview of the current state-of-the-art approaches (post CLIP) in the related works section. Even if this paper uses CLIP as the main architecture (which is fine!) there have been several works since then that propose different pre-training strategies that could be cited. See examples above and from here. https://arxiv.org/pdf/2202.09061 \n2. Add a paragraph that outlines the differences in the contrastive pretraining approach used here vs. these newer pretraining approaches and why the current approach is more favourable. This would make it easier for authors to understand the decisions made and why this architecture might be favourable for this particular data/waveforms as opposed to other architectures.\n3. Add a detailed datasheet for the dataset produced? See the comment in weaknesses above and example datasheets here. https://arxiv.org/abs/1803.09010 \n4. Add either a figure showing readers what the data looks like (e.g., how the waveforms are represented, what a corresponding text snippet is, what demographics are) or a table with details of attributes of data in the section outlining the datasets.\n5. For all results, link in the text where each result is shown. There are several cases in the paper where it is hard to find the table showing results explained in that paragraph and would make it easier to go back to that table (especially since they are all so close to each other in the evaluation section).\n6. Add confidence metrics and resample from the model at least 5 times to assess robustness of model generations.", "broader_impact_concerns": "None.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "exIaoDLjvW", "reviewer": "Reviewer_yd5W", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths:**\n- Demonstrated an effective RAG pipeline for incorporating prior knowledge in text to address the challenge of lacking annotated data (particularly in medical field)\n- Developed a foundation model based on a dataset with 650k paired data items via constrastive learning\n- Carefully evaluated the foundation model (and also a tiny version in view of the possibility of limited computing resources).\n\n**Weaknesses:**\n- The tricks being adopted are inspired from some other works as explained in the paper. The novely regarding machine learning algorithms is limited.\n- While the paper organization and presentaiton is clear in general, some background about some details of ECG (e.g., SCP is not much explained in the manuscript) could be added to further improve the readability for those without bg.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "By arguing that multimodal representation learning is less explored for ECGs and self-supervising solely based on signals is insufficient for exploring the underlying semantic information, this paper proposed a retrieval-augnmented generation (RAG) pipeline to generate a large ECG-text pairs by leveraging knowledge extracted from two books and the LLM. The generated dataset of ECG-text pairs (650k signals) was then used for pretraining based on a particular contrastive learning method. Various ablation studies were performed to evaluate its robustness against a number of training related factors, e.g., the changes in size of pre-training data, degree of ECG-text mis-alignment, etc.\n\nThe key contribution of this work lies in showing the possibilty of multi-modal representation learning by incorporating prior medical knowledge and evaluating how well the pre-trained can perform for the downstream tasks under various settings (zero-shot, few-shot, linear probing and fine-tunng).", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "**Specific comments:**\n- Some typos spotted in Figure 1, e.g., “kownledge”, “accosicates”.\n- Section 3.1.2 provides the general ideas about the use of pre-constructed domain-knowledge database, given information such as standard clinical labels, standard communications protocol for computer-assisted ECG (SCP) statements, diagnostic interpretations, and machine-generated reports associated with ECG data. Some descriptions of the different types of information and how they are stored in the database should be included. Also, more details about how to generate the waveform description should be included.\n- One of the books is “ECG Workout: Exercises In Arrhythmia Interpretation by Huff”. Will this source of the information be biased for the downstream task?\n- PTB-XL and Chapman-Shaoxing are included for pre-training and evaluation, and MMIC-IV-ECG is used only for pre-training. What happens if only MIMIC-IV-ECG is used for pre-training? Under this setting, it will be interesting to see how the zero-shot and linear-probing performance on the two other datasets will be affected. Alternatively, can some of the MIMIC-IV-ECG data be hold out for evaluation as well?\n- The sizes of the three datasets are very different, and MIMIC-IV-ECG obviously dominates the pre-training. Also, is there any class imbalanced situation for the three datasets?\n- According to Table 5a, removing CQA module will lead to performance drop in AUC but not really a big drop. Please contrast the performance gained by incorporating CQA as compared to putting further effort in collecting more data. Under what situations will the performance gain be more obvious?", "broader_impact_concerns": "As publicly available anonymized datasets are used for the study, I do not have concerns on the ethical implications of this work.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "pPU7CY4ntK", "reviewer": "Reviewer_QuP6", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength\n\nS1: Using LLM-based RAG pipeline to enhance the text descriptions makes senses and aligns with researches that use LLM to reduce labeling efforts. \n\nS2: The model design follows SOTA methods and yields good experiment results.\n\nS3: The experiment results are comprehensive.\n\nS4: The presentation is clear and easy to follow.  \n\n============\n\nWeakness\n\nW1: Please report standard deviation for accuracy by running several random initializations (e.g., 10). Currently, the accuracy improvement is marginal in some cases (which is OK), and it is unclear whether these improvements are caused by randomness. \n\nW2: The core contribution of the paper, i.e., LLM-based RAG pipeline, is not explored in depth. For instance, vector search adopts a top-k interface, and how does the value of k affect accuracy? Currently, two textbooks are used as the knowledge base for vector query, will the accuracy be affected if more or fewer textbooks are used? Is it possible that the textbooks return in-consistent contents for the same vector query? How does the authors prompt the LLM? Do different ways of prompting the LLM affect accuracy?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper studies the multi-modal pretraining for ECG representation models. It considers waveforms and their text descriptions, and the original text descriptions of the waveforms are enhanced with an LLM-based RAG pipeline. The multi-modal training combines a contrastive loss and a captioning loss. Experiment results show that the proposed method improves the accuracy of existing approaches for downstream tasks. \n\nThe main contribution of the paper is using an LLM-based RAG pipeline to enhance the text descriptions.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I think the authors have shown that using LLM-based RAG pipeline to enhance the text descriptions works. For a revision, the authors should make the following changes to address the weakness.\n\nR1: Report standard deviation for the accuracy results.\n\nR2: Thoroughly explore the design space of LLM-based RAG and do a better job in justifying the designs.", "broader_impact_concerns": "The paper uses machine learning for medical purposes and may benefit public health. I can see no negative impacts for this paper.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers generally agree that this is solid empirical work with clear writing and convincing results. The major drawback is the lack of more novel methodology; however, for a contribution in a specialized domain like ECG, I think this is okay. Other reviewers' comments are addressed in the rebuttal.
Decision: Accept (poster)