Boosting Masked ECG-Text Auto-Encoders as Discriminative Learners
Authors: Manh Pham Hung
Aaqib Saeed
Dong Ma
Venue: ICML 2025 poster (2025)
Keywords: Cardiovascular Diagnostics
ECG-Text Multi-modal Representation Learning
Contrastive Masked Auto-Encoders
Primary area: applications->health_medicine

Abstract: The accurate interpretation of Electrocardiogram (ECG) signals is pivotal for diagnosing cardiovascular diseases. Integrating ECG signals with accompanying textual reports further holds immense potential to enhance clinical diagnostics by combining physiological data and qualitative insights. However, this integration faces significant challenges due to inherent modality disparities and the scarcity of labeled data for robust cross-modal learning. To address these obstacles, we propose D-BETA, a novel framework that pre-trains ECG and text data using a contrastive masked auto-encoder architecture, uniquely combining generative and boosted discriminative capabilities for robust cross-modal representations. This is accomplished through masked modality modeling, specialized loss functions, and an improved negative sampling strategy tailored for cross-modal alignment. Extensive experiments on five public datasets across diverse downstream tasks demonstrate that D-BETA significantly outperforms existing methods, achieving an average AUC improvement of 15% in linear probing with only one percent of training data and 2% in zero-shot performance without requiring training data over state-of-the-art models. These results highlight the effectiveness of D-BETA, underscoring its potential to advance automated clinical diagnostics through multi-modal representations.

Boosting Masked ECG-Text Auto-Encoders as Discriminative Learners
Hung Manh Pham
Aaqib Saeed
Dong Ma'
Abstract
across specific anatomical regions. The standard 12-lead ECG, which captures unique electrical potential differences from each lead, plays vital role in diagnosing a wide spec- trum of cardiac conditions (e.g. arrhythmias). In recent years, significant progress has been made in leveraging deep learning techniques for automated ECG interpretation (Yan et al., 2019; Ebrahimi et al., 2020; Siontis al., 2021). How- ever, these supervised deep learning approaches often neces- sitate large volumes of expertly annotated data, which are frequently scarce and expensive to acquire. Self-supervised learning (SSL) has emerged as compelling alternative, offering the potential to learn robust representations from abundant unlabeled ECG data. These learned representa- tions can be effectively utilized for zero-shot learning on novel tasks and adapted via fine-tuning specific down- stream applications, thereby mitigating the reliance tensive labeled datasets.
The accurate interpretation of Electrocardiogram (ECG) signals is pivotal for diagnosing cardio- vascular diseases. Integrating ECG signals with accompanying textual reports further holds im- mense potential to enhance clinical diagnostics by combining physiological data and qualitative insights. However, this integration faces signifi- cant challenges due to inherent modality dispar- ities and the scarcity of labeled data for robust cross-modal learning. To address these obstacles, we propose D-BETA, a novel framework that pre-trains ECG and Text data using a contrastive masked Auto-encoder architecture, uniquely com- bining generative and Boosted Discriminative ca- pabilities for robust cross-modal representations. This is accomplished through masked modality modeling, specialized loss functions, and an im- proved negative sampling strategy tailored for cross-modal alignment. Extensive experiments on five public datasets across diverse downstream tasks demonstrate that D-BETA significantly out- performs existing methods, achieving average AUC improvement of 15% in linear probing with only one percent of training data and 2% in zero- shot performance without requiring training data over state-of th-the-art models. These results high- light the effectiveness of D-BETA, underscoring its potential advance automated clinical diag- nostics through multi-modal representations.
Numerous studies have explored the potential of SSL in the ECG domain, demonstrating its efficacy learning repre- sentations from vast quantities of unlabeled data. These efforts generally fall into two main tracks: contrastive and generative approaches. Contrastive methods, exemplified by works such as (Chen et al., 2020; 2021; Chen & He, 2021; Grill et al., 2020; Kiyasseh et al., 2021; Oh et al., 2022; McKeen et al., 2024), aim to learn discriminative rep- resentations by maximizing the similarity between positive pairs (c.g., different augmentations of the same ECG signal) and minimizing the similarity between negative pairs (e.g.. ECGs from different patients) within the embedding space. Conversely, generative approaches (Hu al., 2023; Zhang et al., 2022a; 2023) focus reconstructing the input data, typically by predicting masked or missing segments of the ECG signal, thereby learning to capture the underlying data distribution. Therefore, integrating both contrastive and generative approaches within a unified framework could leverage their complementary strengths, leading more powerful method for learning robust ECG-text representa- tions (Kim et al., 2021: Li et al., 2022b; Song et al., 2024).
1. Introduction
Electrocardiograms (ECGs), obtained through non-invasive electrode placement, provide critical window into the heart's electrical activity by measuring voltage differences
SSingapore Management University Bindhovee Univer- sity of Technology. Correspondence to: Dong Ma ddongma@smu..duu
Despite advancements, existing ECG-based SSL approaches have largely overlooked the valuable information embedded within clinical text reports, which offer key insights into underlying cardiac conditions and have the potential to sig- nificantly enhance a model's diagnostic accuracy (Zhang et al., 2022c; Chen et al., 2022). This oversight highlights
Proceedings of the 4nnd International Conference on Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).
OOr code and checkpoint made available https:// githus com/ wtt./com/mmmmmca /tt////nnhp2211111m 1012211111BB11 1/D-BRTA, BRTA.

---

Boosting Masked ECG-Text Auto-Encoders as Discriminative Learners
critical gap in the field: the lack of emphasis on jointly learn- ing ECG-text cross-modal representations. Some recent efforts (Liu et al., 2024b; Lalam et al., 2023; Li et al., 2024; Liu et al., 2024a; Yu et al., 2024) have attempted to bridge this gap by integrating ECG signals and clinical reports through cross-modal contrastive learning, mainly employing relatively standard encoder models (e.g.. ResNet (He et al., 2016), Bert (Devlin et al., 2019)). This makes the poten- tial of learning robust representations that capture the intri- cate interplay between ECG signals and their corresponding textual descriptions, shown in generative approaches re- main largely unexplored. Moreover, the prevailing reliance on these contrastive methods presents inherent limitations. They depend on the availability of negative samples and of- ten struggle to capture cross-modal relationships effectively due to difficulties in defining appropriate negative pairings across different modalities.
recent years.
Early contrastive methods such as SimCLR (Chen et al., 2020), MoCo (Chen et al., 2021), SimSiam (Chen & He, 2021). and BYOL (Grill et al., 2020) introduced the concept of maximizing agreement between augmented views of the same data sample by employing augmentation strategies to create challenging positive and negative pairs. In the context of ECG signals, recent approaches like 3KG (Gopal et al., 2021) apply physiologically inspired spatial and tem- poral augmentations, using vectorcardiogram (VCG) trans- formations to capture the three- dineessiinnii spatiotemporal characteristics of the heart's electrical activity. Similarly, CLOCS (Kiyasseh et al., 2021) developed Contrastive Multi- Segment Coding (CMSC), which enhances the model's ability to handle varying ECG signal characteristics across different axes-space, time, and patients. Building on this, (Oh et al., 2022) incorporates Wav2vec 2.0 (Baevski et al., 2020), CMSC, and random lead masking to simulate dif- ferent global and local lead configurations during training, thereby improving model robustness and achieving impres- sive results on ECG downstream tasks.
In this work, we depart from relying solely on contrastive learning or stand-alone generative approaches for cross- modal representation learning. We introduce D-BETA, a novel hybrid framework that synergistically integrates both learning paradigms to effectively capture fine-grained input details and discriminative ECG-text features. Our approach encoder specifically for ECG
On the other hand, generative approaches (Hu et al., 2023; Zhang et al., 2022a; Na et al., 2024) are less prevalent, but play a crucial role in ECG SSL. These methods focus on capturing the underlying structure of the data by train- ing auto-encoder models to generate or reconstruct masked input data, enabling the model to understand and repre- sent key features and patterns. For instance, ST-MEM (Na et al., 2024) utilizes a masked auto-encoder with spatio- temporal patchifying technique to model relationships in 12- lead ECG signals. Additionally, the Cross- Reconstruction Transformer (CRT) (Zhang et al., 2023) employs frequency- domain and temporal masking to reconstruct missing ECG segments, demonstrating the innovative use of generative SSL in ECG analysis.
employs a transformer-basee signals and a rell-pre-trained language model for clinical text encoder in masked auto-encoder architecture, together with tailored loss functions that promote the joint learning of robust cross-modal representations. Additionally, we introduce a nearest-neighhor negative sampling strategy, a crucial refinement often overlooked in previous methods, to ensure that negative samples are contextually selected and thereby, enhance the discriminative capability of the learned representations. To rigorously evaluate the efficacy of D-BETA, we conduct extensive experiments on vari- ous public ECG datasets and demonstrate that our method significantly outperforms recent state-of-the-at baselines across all evaluation settings.
ECG-Text Multi-modal Representation Learning. Multi-modal representation learning combines information from different data types, shown to effectively improve model performance (Lin al.,, 2024; Du et al., 2023). Partic- ularly, pioneering works like CLIP-based models (Radford et al., 2021; Rasheed et al., 2023; Zhai et al., 2023) have proven the power of contrastive learning in aligning visual and textual modalities, achieving strong generalizations across a broad range of tasks. Applying similar ideas to the ECG domain, recent efforts (Lalam et al., 2023; Yu et al., 2024; Liu et al., 2024a:b) show promising progress in the field. Among them, MERL (Liu et al., 2024b) leverages cross-modal and uni-modal alignment techniques together with test-time clinical knowledge enhancement, which notably generalizes ECG and text-based medical zero-shot classification tasks. However, they often utilize genetic architectures (e.g., ResNet ECG encoder, Bert-based
2. Related Work
ECG Self-uppervsed Learning, Self-supervised learning (SSL) has been shown to work effectively across various modalities, including vision (Li et al., 2022a; Han et al., 2021), language (Devlin et al., 2019; He et al., 2020; Chung et al., 2024), and time-series data (Tonekaboni et al., 2021; Zhang et al., 2022b; Saced et al., 2019). Particularly, recent advances in applying SSL to ECG signals have demon- strated that models can learn meaningful representations from large amounts of unlabeled data, which is crucial in medical domains where labeled datasets are often limited and expensive to acquire. Here, we discuss two common SSL approaches: generative and contrastive, which have seen notable progress in ECG representation learning in
2

---

Boosting Masked ECG-Text Auto-Encodee as Discriminative Learners
text encoder) and especially overlook the critical role of negative sample selection for contrastive learning and lacks exploring generative approaches for fine-grained multi-modal learning, limiting performance in end tasks.
inally developed for efficiently processing sequential data in parallel (Vaswani et al., 2023). We first follow (Oh et al., 2022) to apply a masking strategy to the ECG in- put X E R Lx to encourage robust feature learning, where I is the length of the signal and C' is the number of channels. Specifically, we leverage random lead masking as an on-the- fly augmentation where each lead randomly masked with probability of p = 0.5 during pre-training. Furthermore, we use dropout layer the input with p = 0.1 to enable masking modeling. We then pass the masked input into series of convolutional layers, each followed by GELU activation functions and group normalization. The extracted features are subsequently projected into a -os-dimensional space. Following that, we add a convolutional positional encoding layer to preserve the temporal order of the ECG sequence. Next, we employ eight transformer encoder lay- ers, each including a multi-head self-attention mechanism that allows the model to attend to different parts of the input sequence simultancously. We conduct an experiment explor- ing the effects of different numbers of transformer layers in Section 4.3.
3. Method
We propose D-BETA, framework designed learn gener- alizable cross-modal representations by aligning ECG sig- nals and corresponding medical text reports. D-BETA lever- ages masked language modeling (MI.) and masked ECG modeling (MEM) to reconstruct randomly masked segments within the input text and ECG signals, respectively. This encourages the model to learn fine-grained features within each modality. Furthermore, we introduce ETS (ECG-Text Sigmoid) loss, as inspired by SigLIP (Zhai al., 2023), and nearest-neighhor negative sampling strategy. These di- rectly promote discriminative representation learning and en- hance modal alignment, besides the ECG-text match- ing (ETM) learning task.
Figure I depicts the overall architecture of D-BETA, which comprises two main branches. The ECG encoder utilizes a transformer- bassed architecture (Vaswani al., 2023) to pro- cess the input ECG signals and generate corresponding rep- resentations, denoted as H2 E R' where L, represents the sequence length of the ECG signal and d represents the emhedding dimension. The text encoder utilizes the recent pre-trained Flan-T5 model (Chung et al., 2024) which, to our knowledge, has not been previously applied this ECG domain, extract high-level semantic embeddings from the clinical text, denoted as H, RLixd, where L, repre- sents the sequence length of the text. These encoder outputs are then passed through fusion module. which employs a cross-attention mechanism to integrate information from both modalities, generating fused representations denoted as H E RL)x+ The model subsequently employs three distinct heads: two decoders, responsible for recon- structing the masked portions of the ECG signal (X) and text (Tn), respectively, and contrastive prediction head for ECG-text matching. Additionally, we introduce two projection heads, and gt. following the ECG and text en- coders, respectively. These projection heads, along with the ETS loss, facilitate learning discriminative representation between these modalities. The model is trained by jointly optimizing four loss functions: masked language modeling loss (CMLM), masked ECG modeling loss (LMEM), ECG- text matching loss (LetMs), and the ETS loss (LETS) ). The subsequent subsections provide a detailed description of each component within the D-BETA framework.
Text Encoder. For our text encoder, we utilize the Flan- T5-base encoder (denoted as Fo), which outputs 768- dimensional embeddings. The input to the encoder con- sists of token indices generated by the Flan-T5 tokenizer, represented as T E ZM, where M is the maximum se- quence length. Flan-T5 is an advanced version of the T5 model (Raffel et al., 2023), which has been pre-trained on a massive and diverse text dataset covering numerous tasks, such as summarization and question answering. Note that our text encoder is fine-tuned during the pre-training stage. In our attempt to demonstrate the effectiveness of recent Flan-Ty in the ECG-domain- -based context, we also conduct an ablation with various text encoders in Section 4.3.
Fusion Module. The fusion module begins with linear projections that map the outputs of the ECG and language encoders to f68-dimensional space. We apply modality- specific embeddings to the projected features distinguish between ECG and text data. Importantly, we employ cross- attention to integrate the ECG and textual information, al- lowing each modality to inform the other by learning the relevant features. This cross-attention mechanism is cru- cial as it enables the model to leverage the complementary strengths of both ECG and text data more effectively.
Decoders and Self-Supervised Tasks. After the fusion module, three network heads are introduced, each associated with a specific task or loss function: masked language mod- eling (MLM), masked ECG modeling (MEM), and ECG- text matching (ETM). MLM and MEM are designed for reconstruction tasks, while ETM adopts a contrastive leamn ing approach to align the different modalities. We detail each task and its corresponding loss function below:
3.1. Multi-Modal Masked Auto- EecodeEcs..ss.
ECG Encoder. We implement the ECG encoder (denoted
as F,) based on a transformer architecture, which was orig-
3

---

Boosting Masked ECG-Text Auto-Encoders as Discriminative Learners
12-leads ECG Signals
Fx
X1
ECG
ECG
MEM Loss
Encoder
Decoder
Fuslon Fusion Module
Cross-attention
9x
ECG-ETS Head
ETS Loss
Zzpky
ETM Head
ETM Loss
Text-ETS Head
Textual Clinical Report
gt
fibrtiiatoon changes may be due to myocardial ischemia. repolariation changes partly due rhythi, abnormal ecg.
f
Text
Text Decoder
MLM Loss
N35
Encoder
Ft
Figure J. Illustration of our contrastive masked ECG- ECG-language modeling technique.
Masked Language Modeling (MLM). The MLM task sists of a dense layer that outputs probability distribution over the vocabulary. The MLM head focuses on predicting the masked tokens the input text sequence, encouraging the model to learn contextualized word embeddings through a reconstruction task. We use the cross-entropy (CE) loss for MLM, as shown in Equation 1:
and MEM. This is formulated binary classification task, where the ETM task's head consists of a single dense layer that outputs scalar &x,tt, representing the predicted prob- ability. The ETM loss defined as the binary cross-entropy loss:
LETM
1
Σ
yk log (2xa,te)
B
k=
[MLM =
1 B
Σ
Σ
log
(1)
(1 - 1og( (1-11111 o(xx,
(3)
j=1 MEM,
where B represents the batch size, M, is the of masked positions in the jth sequence, fn is the masked token at position in the jth sequence, tj,,, represents the 5th input sequence with masked tokens removed, and 0 represents the model parameters.
where is the sigmoid function, y/ if (xk; tx) is positive pair, and yı: 0 otherwise.
3.2. ECG-Text Discriminative Learners.
ETS Loss Function. In multi-modal masked encoder architectures such as (Chen al., 2022), contrastive learning's 's effectiveness can be limited by the inherent ten- sion between the reconstruction-focee autoencoders and the discriminative nature of contrastive learning. They are more biased for learning to reconstruct masked inputs in generative manners. This can hinder the model's capability learn discriminative features useful for downstream tasks, such zero-shot inference or linear probing. Furthermore, although the ETM loss in such ar- chitectures can serve form of contrastive loss. it may not be sufficient for building a robust ECG encoder. Specif- ically. the ETM module is primarily designed for binary classification based on fused features rather than directly enhancing the discriminative power of individual encoders. This limitation restrict the model's ability to produce high-quality multimodal embeddings.
Masked ECG Modeling (MEM). Similar to MIM, the MEM task aims to reconstruct the masked ECG inputs. It consists of linear embedding layer that maps the input sequence to space (384), followed by learnable mask tokens that represent the missing portions of the sequence. We apply positional encoding preserve the temporal struc- multi-layer
generative tasks of
ture of the ECG data. Subsequently, transformer decoder to model the dependencies within the sequence. Finally, a linear projection layer produces the predicted ECG signal (&;). We train the MEM head using the mean squared error (MSE) between this predicted signal and the ground truth signal (x;), as shown Equation 2:
LMEM
1 B
Σ
|
x
ECG-Text Matching (ETM). Finally, we use ETM to pro- mote alignment between ECG signals and their correspond- ing text reports, which further supports the fused feature space learning, together with generative aspects from MLM
Therefore, we propose strengthening discriminative aspect in multi-modal masked auto-encoder architectures using ETS loss, as shown formula This approach avoids the costly global normalization of fettmx--based contrastive

---

Boosting Masked ECG-Text Auto-Encoders as Discriminative Learners
12-lead ECG
Flan-T5 Feature Space
Possible Mapping Texts
Positive
changes nonspecific
right bundle branch block
Negative
sinus rhythm, normal ECG
FAISS
rbbb with left anterior fascicular block. lateral wave changes are nonspecific, abnormal ecg
sinus rhythm. rar (vl) probable
normal variant. normal ECG
Figure 2. Illustration of N3S in selecting negative samples within Flan-T5 space.
losses by operating independently on each ECG-text pair (sigmoid-base basedd, improving memory efficiency and scalabil- ity. We introduce two additional network heads to the ECG and text encoders, respectively. Each head consists of a pooling layer, Tanh activation function, and a dense layer, enabling them to output 768-dimensional embeddings (de- noted x', T 768 for the tt ECG sample and t'y E 768 for the jth text report).
To perform this process, we employ FAISS (Facebook Al Similarity Search) (Douze et al., 2024), high-performance library designed for indexing and searching large collections of dense vectors. FAISS allows us to apply N3S to large- scale datasets in a computationally tractable manner.
4. Experiments
4.1. Implementation Details.
1 B
1
4.1.1. PRE-TRAINING TASK.
LETS
log 1 e -
(4)
Pre-train Dataset. In the pre-training stage, utilize the MIMIC-IV-ECG v1.0 database (Gow et al., 2023), which in- cludes 800,035 paired samples derived from 161,352 unique subjects. This dataset contains numerous 10-second ECG recordings sampled at 500 Hz and the corresponding text reports. Each ECG recording will have several reports, and we simply merge them into one single report (diagnosis). We apply some necessary processing steps prepare the custom dataset for training (e.g., remove empty contain- ing NaN ECG recordings and clean text by using lowercase, strip, and punctuation removal), which eventually yields a training size of 779891 samples. We provide representative examples of ECG-text pairs in Appendix A.
1=1
where yij = 1 for positive (matching) ECG-text pairs, and V/y otherwise.
Nearest-neighhor-baeed Negative Sampling (N3S). In contrastive learning, the selection of negative samples signif- icantly impacts the training process (Xu et al., 2022). Con- ventional methods often employ random sampling, where negative text reports are chosen randomly to replace positive texts. However, this approach may lead to false negative selection, especially in medical datasets, where randomly chosen reports might share substantial similarities with the positive reports, hindering effective contrastive learning. This is discussed further in the Appendix A.2.
Experimental Configurations. For model training, we use the Adam optimizer with learning rate of x 10 and use a tri-stage scheduler with ratios of 0.1, 0.4, and 0.5 for learning rate adjustments. The optimizer is configured with 81 0.9, 82 = 0.98, epsilon value of 1 x 10-6, and a weight decay of 0.01. We pre-train the proposed model for 300000 steps, maintaining a batch size of 128. The quantitative experiments are conducted on a single NVIDIA H100-80GB GPU.
Therefore, propose neirest-neighhor negative sampling (N3S), which selects negative samples based on their dis- similarity in the Flan-T5's feature space (Figure 2), ensuring they are sufficiently distinct from the positive samples while remaining semantically related to the domain. Specifically, we first utilize mraitraned Flan- T5 (small) to generate vector representations, denoted as V, R512, for each text report t in the training dataset Dtrain. These embeddings cap- ture the semantic meaning of the reports. During training, for a given ECG and its corresponding positive text report (Nx, tt) in half of the training batch B, the negative report if is selected as one of the top 64 largest cosine distance reports from the positive report's 's embedding Veg. As the training progresses with batches being updated randomly, the negative samples continually change, introducing vari- ability while maintaining domain relevance.
4.1.2. DOWNSTREAM TASKS.
Downstream Datasets. We evaluate pre-trained en- coders on five widely-used public datasets: PhysioNet 2021 (Reyna et al., 2021), PTB-XL (Wagner et al., 2020), CSN (Zheng et al., 2022), CPSC2018 (Liu et al., 2018), and CODE-test (Ribeiro et al., 2020). We summarize the key information of each dataset as follows:
5

---

Boosting Masked ECG-Text Auto-Encoders as Discriminative Learners
Table Performance for 5 lead combinations in diagnosis classification (Dx., by CinC scores scaled by 100) and patient identification (Id., by %6. P-N-lead indicates N zero- padded unavailable leads.
zero-padded
# Leads
Methods
Tasks
12-lead
P-6-lead
P-3-lead
P-2-lead
P-I-lead
Dx.
71.4
64.3
67.6
61.1
52.5
W2V (Baevski et al., 2020)
ld.
49.2
41.1
47.0
41.4
24.7
Dx.
62.5
52.2
57.5
50.7
40.6
CMSC (Kiyasseh 2021)
Id.
51.3
39.2
51.0
37.8
22.7
Dx.
60.0
$1.5
56.3
50.5
41.8
3KG (Gopel et al., 2021)
ld.
32.0
36.7
31.0
19.8
SinCLRRRLLM) (Chen et al., 2020)
Dx.
57.8
49.7
53.5
48.4
39.3
ld.
35.3
28.9
36.8
30.4
19.2
W2V+CMSC (Oh et al., 2022)
Dx.
58.6
48.2
Id.
55.0
43.7
46.6
41.0
28.0
W2V+CMSC+RLM eO at. 2022)
Dx.
73.2
66.2
71.4
65.6
55.4
ld.
57.7
45.9
54.8
45.7
31.3
Dx.
85.7
81.1
84.2
81.9
76.5
D-BETA
ld.
57.7
Table 2. Performance comparison (AUC in %) across multiple methods and datasets. The results are shown for different percentages of training data used (1%, 10%, 100%).
Super
PTBXL-Sub
PIBXL-Form
-hyyhm
CPSC2018
CSN
Methods
1%
10%
100%
1%
10%
100%
1%
10%
100%
1%
10%
100%
1%
10%
100%
1%
10%
100%
SimCLR (Chen et al., 2020)
63.41
69.77
73.53
60.84
68.27
73.39
54.98
56.97
62.52
51.41
69:44
77.73
59.78
68.52
76.54
59.02
67.26
73.20
BYOL (Gril al.,
71.70
73.83
76.45
57.16
67.44
71.64
61.63
74.40
77.17
60.88
78.75
54,20
71.92
74.69
BarlowTwins (Zbontar et al., 2021)
72.87
75,96
78.41
62.57
70.84
74.34
52.12
60.39
66.14
50.12
73.54
77.62
55.12
72.75
78.39
60.72
71.64
77.43
MoCo-v3 (Chen et al., 2021)
73.19
76.65
78.26
55,88
69.21
76.69
50.32
63.71
71.31
51.38
71.66
74.33
62.13
76.74
75.29
54.61
74.26
77.68
SimSlam (Chen & He, 2021)
73.15
72.70
75.63
62.52
69.31
76.38
55,16
62.91
71.31
49.30
69.47
75.92
58.35
72.89
75.31
58.25
68.61
77.41
TS-TCC (Eldele et al, 2021)
70.73
75.88
78.91
53.54
66.98
77.87
48.04
61.79
71.18
43.34
69.48
78.23
$7.07
73.62
78.72
55.26
68.48
76.79
CLOCS (Kiyassch et 2021)
68.94
73.36
76.31
57.94
72.55
76.24
$7.79
72.65
76.31
59.59
77.78
77.49
54.38
71.93
76.13
ASTCL (Wang et al., 2023)
72.51
77.31
81.02
61.86
68.77
76.51
44.14
60.93
66.99
52.38
71.98
76.05
57.90
77.01
79.51
56.40
70.87
75.79
(Zhang et al., 2023)
69.68
78.24
77.24
61,98
70.82
78.67
59.49
68.73
73.52
74.41
58.01
82.03
56.21
73.70
78.80
ST-MEM (Na et al., 2024)
61.12
66.87
71.36
54.12
57.86
63.59
55.71
59.99
66.07
51.12
65:44
74.85
56.69
63.32
70.39
59.77
66.87
71.36
MERL (Liu et al., 2024b)
82.39
86.27
88.67
64.90
80.56
84.72
58.26
72.43
79.65
53.33
82.88
88.34
70.33
85.32
90.57
66.60
82.74
87.95
D-BETA
83.15
88.36
90.11
77.74
82.92
85.15
70.10
78.91
83.98
16.66
92.83
96.71
88544
91.35
94.92
80.04
87.36 90.71
PhysioNet 2021. This contains ECG samples (500 Hz) ranging between 5 and 144 seconds. We process and fine- tune the subsets as described in (Oh et al., 2022) to validate the pre-trained ECG encoder in two downstream tasks: 1)
CPSC2018. The dataset contains 6,877 standard 12-lead
ECG recordings (500 Hz), which cover 9 distinct categories. Similarly, we also use the same data configuration follow- ing (Liu et al., 2024b).
26-multi-label cardiac arrhythmia classification (Dx.); 2) patient identification (Id.), predicting patient ownership of ECG recordings
This contains 827 12-lead ECG samples (400 Hz) at varying lengths covering 6 abnormalities, annotated by experienced residents and medical students. We resample the signals to 500 Hz and adjust the lengths to 10 seconds. We provide more detail about this dataset in Appendix A.
CODE-test:
PTB-XI. The PTB-XI. dataset includes 21,837 ECG signals collected from 18,885 patients. Each sample has a 12-lead ECG recording sampled at 500 Hz over 10 seconds and corresponding cardiac labels. We follow (Liu al., 2024b) to split this dataset, including four sub-groups (super, sub, form, and rhythm). We consider them as the four separated datasets and prepare each of them with the same train, val, and test set as in the original paper (Wagner et al., 2020).
Experimental Configurations. To evaluate our model's performance on downstream tasks, we conduct three ex- periments: 1) First, integrating a linear layer on top of the pre-trained ECG encoder and tinettuning the entire model to test its efficacy in two tasks within the Physionet 2021 dataset: Dx. (by CinC score) and ld. (by % accuracy). We report the results with five cases of lead combinations, as presented in (Oh et al., 2022); 2) Second, we also implement a linear classifier but keep the ECG encoder frozen. This linear probing approach is applied at different training set
CSN. This dataset consists of 23,026 ECG recordings sam- pled at 500 Hz for 10 seconds with 38 distinct labels, which also supports the evaluation in a classification task. We use 70%:10%:209 data split as processed in (Liu et al., 2024b).
6

---

Boosting Masked ECG-Text Auto-Encoders as Discriminative Learners
Table 3. Zero-shot performance (AUC in %) comparison across multiple datasets.
Methods
PTBXI-Super
PTBXL-Sub
PTBXI-FForm
PTBXI-Rhythm
CPSC2018
CSN
Averuge
MERL
74.2
75.7
65.9
78.5
82.8
74.4
75.3
D-BETA
76.2
75.9
66.1
88.6
80,1
76.3
77.1
Table 4. Zero-shot performance (AUC in %6) under data distribution shift.
Source Domain
PTBXL-Super
CPSC2018
CSN
Zero-shot
Training Ratio
Target Domain
CPSC2018
PTBXI-Super
CSN
PTBXL-Super
CPSC2018
SimCLR (Chen et al., 2020)
X
100%
69.62
73.05
56.65
66.36
59.74
62.11
BYOL (Grill 2020)
100%
70.27
74.01
57.32
67.56
60.39
63.24
Barlow Twins (Zbontar et 2021)
X
100%
68.98
72.85
35,97
63,89
58.76
61.35
McCo.v3
2021)
SimSiam (Chen & He, 2021)
X
100%
70.06
73.92
$7.21
67.48
60.23
63.09
TS-TCC (Eldele al., 2021)
100%
71.32
58.47
68.
61.55
64.48
CLOCS (Kiyasseh al., 2021)
100%
68.79
72.64
$5.86
65.73
58.69
61,27
ASTCI- al., 2023)
X
100%
69.23
73.18
56.61
66.27
59.74
62.12
CRT (Zhang al., 2023)
100%
70.15
$7.39
67.62
60.48
63.33
ST-MEM (Na et al., 2024)
X
100%
76.12
84.50
62.27
73.19
73.05
64.66
(Liu al., 2024b)
76.56
74.15
D-BETA
0%
72.09
79.11
77.12
82.91
76.24
80.10
Table 5, ECG interpretation comparison (AUC in %): Human experts
DNN (Ribeiro et al., 2020) D-BETA.
Cardio Resident
Emergency Resident
Medical Student
DNN
D-BETA (Zeroshol)
92.07
90.52
93.61
96.59
96.79
sizes (1%, 10%. and 100%) to assess the macro AUC score (%) the PTB-XI,, CSN, and CPSC2018 test datasets, fa- cilitating comparison with our baseline (Liu et al., 2024b); 3) Finally, we investigate zero-shot classification (AUC) PTB-XL, CSN, CPSC2018 and CODE-test datasets. Here, the texts used are obtained by passing the category names into GPT-4o to capture better medical context. The de- tailed configuration on each experiment is mentioned in Appendix A.I.
Linear Probing Evaluation. Table 2 presents the linear probing results, where method demonstrates clear ad- vantage over the baseline approaches. Notably, with only 1% of the training data, our method shows a substantial improve- ment over MERL, especially in CSN (14% enhancement) and Rhyth- (33% datasets. Similarly, impressive results are observed at 10% and 100% of the data. For exam- ple, on the PTBXL-Rhythm dataset, our method achieves approximately 10% improvement the 10% configuration. On the CPSC2018 8 dataset, we also observe considerable increase from 90.57% % to 94.92% when using 100% of the training data.
4.2. Experimental Results.
Full Fine- Tane-TTnngg Evaluation. As shown in Table 1, our method consistently outperforms previous approaches (Oh et al., 2022) in both examined tasks. In the classifi- cation task, our model achieves 85.7% accuracy with all 12 leads, significantly higher than the best baseline (WV+CMSS++LLM), which is 73.2%. This number even lower than our setting with only lead usage (76.5%). Inter- estingly, the e-ad combination yields the highest result, only 1.5% lower than using all leads, while the 2- lead and 6-lead combinations produce comparable results, both around 81.5%. This suggests that the selected leads (I, II, V2) capture sufficient information for accurate perfor- mance. A similar pattern emerges in the identification task, where our model achieves 41.1% accuracy with a single lead, 60.5% with leads, and 65. with 12 leads, surpassing the best baseline by 7%.
Zero-shot Evaluation. We first compare our results and the best results of MERL in zero-shot settings across six datasets, shown in Table 3. On average, method achieves 77%, outperforming MERL by 2%. Notably, MERL performs impressively on the CPSC2018 dataset, while its results on the other five datasets are consistently lower than ours. Next, we extend the comparison of our method with MERL and other SSL baselines (Liu al., 2024b) under data distribution shifts. Specifically, we com- pare linear probing (100% training size) of SSL methods with MERL's and our zero-shot approach. In this setup, the source domain and target domain share some common categories. Details on this implementation can be found in Appendix 1. As shown in Table 4, our results surpass MERL and other SSL methods, except when CPSC2018 is the target domain, which aligns with our previous obser-

---

Boosting Masked ECG-Text Auto-Encoders as Discriminative Learners
vations. Finally, Table 5 shows that our zero-shot model outperforms three experienced cardiologists (over 3%) and also the in-domain model (Ribeiro et al., 2020), i.e., trained with millions of annotated ECG examples ?. We discuss more on zero-shot settings Appendix A.3.
components, including Flan-T5, ETS, and N3S. Specifi- cally, start by eliminating the N3S and train the model with randomly selected negative samples. Subsequently, we take the ETS loss away to assess its effectiveness in capturing rich representative embeddings in both encoders. Lastly, by replacing the Flan-T5 language encoder with a standard Bert-base architecture (Devlin et al., 2019), we consider this as the baseline model. Table demonstrates the results of this experiment. It can be seen that ETS signif- icantly enhances performance, showing an improvement of approximately 15% in both full fine-tuning and linear prob- ing settings over the baseline model. Meanwhile, adding N3S improves zero-shot classification by 2%, and introduc- ing Flan-T5 enhances performance linear probing by 4% compared to the baseline. These results underscore the ef- fectiveness of each component in optimizing the model's performance.
To better understand how our method improves downstream performance, we visualize and compare the t-SNE embed- dings generated by our ECG encoder on the CSN test set with those from MERL. For clearer visualization, we in- clude only samples from unique categories and exclude categories with fewer than 50 samples. Figure 3 reveals that our embeddings show more well-defined and distinct clusters representing different ECG diagnoses.
SA
ALS
APB
AF
SVT
TWC
ST
MERL
Ours
Text Encoders. In this ablation study, we evaluate the performance of four pre-trained language models, namely Bert (Devlin et al., 2019), Deberta (He et al., 2020), Med- CPT (Jin et al., 2023), and Flan-T5 (Chung et al., 2024) to determine the most suitable language encoder for our model. Here, only the base versions were tested. As shown in Table 7, Flan-T5 outperforms the others across multiple metrics, highlighting the importance of choosing model that excels not only in general text processing but also in capturing Nomain-specinf nuances, such as ECG reports.
Figure 3. T-SNE visualization on the CSN test set.
4.3. Ablation Studies.
We evaluate the impact of the key model components, the choice of language encoders, and varying the number of transformer layers in the ECG encoder for ablation stud- ies. Here, we focus on three downstream tasks, including full fine-tuned diagnosis classification (results across five lead combinations), linear probing at 1% training size, and zero-shot classification (results across PTB-XL, CSN, and CPSC2018 8 datasets) using category name 3.
Table Effects of different text encoders.
Text encoder
Full fine-tune Linear probing
Zero-shot
Fla-I1
81.88 + 3.52
80.52 :: 6,08
72.50:: 9.01
Med-CPT
81.02 + 3.61
79.57
6.32
71.81 + 9.14
Deberta
79.23 + 3.65
78.24 = 6.21
70.67 + 9,88
Bert
78.08 1. 3.91
77.58 6.49
69.14 9.97
Table 6, Effects of the model components: ETS Loss, N3S.
! Flan-T5, 2
Number of ECG Transformer Layers. As part of our ab- lation study, we explore the impact of varying the number of transformer layers (1, 4, 8) in the ECG encoder. As shown in Table 8, increasing the number of layers significantly improves performance. Specifically, the 1-layer model per- forms 11% worse than the 8-layer model (proposed) in full fine-tuning and 13% worse in linear probing. For zero-shot, the 8-layer model still delivers superior results, with 2% and 3% higher performance than the 4-layer and 1-layer models, respectively. Although these differences are smaller than in full fine-tuning, they still highlight the ECG encoder design's 's impact on improving performance. is also worth noting that this same zero-shot setting (without GPT-4o's support), even our I-layer case outperformed MERL by ap- proximately 8%, achieving approximately 70% compared to MERL's 62%.
fime-tume
probing
Zero-shol
81.88
3.52
80.52 + 6.08
72.50 9.01
80.93 + 3.74
78.29 + 6.19
70.61 = 8.10
78.29 3.87
67.19 6.14
76,81 3.96
63,50 1 6.95
Effects of Key Components in D-BETA We system- atically remove one component at time from the default proposed model assess the contribution of different model
2 Medical students outperform residents due to recent frequently focused training, aligning with prior work (Ribeiro et al., 2020).
"e do not use GPT-4o for context enhancement, as our objec- tive provide the impact of our proposed components in the context of ablation studies.
8

---

Boosting Masked ECG-Text Auto-Encoders as Discriminative Learners
self-supervised vision transformers. In Proceedings of the IEEE//VV international conference on computer vision, pp. 9640-9649, 2021.
Table Rffects of number of transformer layers in ECG encoder.
8 Layers
fine-tune
Linear probing
Zero
8
81.88
3.52
80.52 6.08
72.50-1 9.01
4 77.63 # 4.14
70.17 + 7.60
70.64 = 8.63
Chen, Z., Du, Y. Hu, J., Liu, Y., Li, G., Wan, X., and Chang, T.-H. ical vision-and and-language pre-training. In International Conference on Medical Image Computing and Computer- Assisted Intervention, pp. 679-689, Springer, 2022.
4.55
66.83 7.52
9.51
masked autoencoders for med-
5. Conclusion
We propose D-BETA. novel contrastive masked transformer-based architecture to pre-train ECG signals and corresponding texts. Our approach is generative self- supervised learning, enhanced with ETS loss, and nearest- neighbor negative sampling to support contrastive aspects. Experimental results demonstrate that our method outper- forms previous approaches across multiple datasets and wide range of downstream tasks with over 100 cardiac conditions. D-BETA shows promise in advancing ECG- based diagnostic models, paving the way for more accurate, efficient, and personalized cardiac care.
Chung, H. W. Hou, L., Longpre, S., Zoph, B., Tay, Y., Fe- dus, W., Li, Y., Wang, X., Dehghani, M., Brahma, S., et al. Scaling instruction finettunfnenddd language models. Journal of Machine Learning Research, 25(70):1-53, 2024.
Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. Bert: Pre-training of deep bidirectional transformers for lan- guage understanding. In Proceedings of the 2019 confer- ence of the North American chapter of the association for computational linguistics: human language technologies, volume 1 (long and short papers), pp. 4171-4186, 2019.
Douze, M., Guzhva, A., Deng, C., Johnson, J., Szilvasy, G., Mazaré, P.-E., Lomeli, M., Hosseini, L., and Jégou, H. The faiss library. 2024.
Acknowledgments
This research was supported by the Google South Asia & Southeast Asia research award 2024.
Du, C., Teng, J., Li, T. Liu, Y., Yuan, T., Wang, Y., Yuan, Y., and Zhao, H. On uni-modal feature learning in supervised multi-modal learning, 2023. URL https://aras erg/abst /2005. 01233.
Impact Statement
Our D-BETA framework advances ECG diagnosis by lever- aging multi-modal self-supervised learning, delivering ro- bust benchmarking results across various downstream tasks. We hope D-BETA provides valuable modeling insights for ECG diagnosis and contributes to the healthcare commu- nity focused on cardiovascular diseases. We also publicly release the pre- pre-trained models and code, empowering future research and extending ECG analysis to broader clinical applications.
Ebrahimi, Z., Loni, M., Daneshtalab, M., and Gharehbaghi, A. A review on deep learning methods for eog arrhythmia classification. Expert Systems with Applications: X, 7: 100033, 2020.
Eldele, E., Ragab, M., Chen, Z., Wu, M., Kwoh, C. K., Li, X., and Guan, C. Time-series representation learning via temporal and contextual contrasting. In Proceedings of the Thirtieth International Joint Conference Artificial Intelligence, IJCAI-21, pp. 2352-22222222 2359, 2021.
References
Gopal, B., Han, R., Raghupathi. G., Ng, A., Tison, G., and Rajpurkar, P. 3kg: Contrastive learning of 12-lead elec- trocardiograms using physiologicalll-inspii tations. In Machine Learning for Health, pp. 156-167. PMLR, 2021.
Baevski, A., Zhou, Y., Mohamed, A., and Auli, M. wav2vec 2.0: A framework for self-supervised learning of speech representations. Advances in neural information process- ing systems, 13:1244--22460, 2020.
augmen-
Chen, T., Kornblith, S., Norouzi. M., and Hinton, G. A simple framework for contrastive learning of visual rep- resentations. In International conference on machine learning, pp. 1597-1607. PMLR, 2020.
Gow, B., Pollard, T., Nathanson, L. A., Johnson, A., Moody, B., Fernandes, C., Greenbaum, N., Berkowitz, S., Moukheiber, D., Eslami, P., et al. Mimic-iv-ecg- diagnostic electrocardiogram matched subset. Type: dataset, 2023.
Chen, X. and He, K. Exploring simple siamese represen- tation learning. In Proceedings of the IEEE/CVF con- ference on computer vision and pattern recognition, pp. 15750-15758, 2021.
Grill, J.-B., Strub, F., Altché, F., Tallec, C., Richemond, P., Buchatskaya, E., Doersch, C., Avila Pires, B., Guo, Z., Gheshlaghi Azar, M., et al. Bootstrap your own latent-a new approach self-supervised learning. Advances in
Chen, X., Xie, S., and He, An empirical study of training

---

Boosting Masked ECG-Text Auto-Encoders as Discriminative Learners
neural information processing systems, 33:21 2020.
Li, Liu, C., Cheng, S., Arcucci, R., and Hong. S. Frozen language model helps eog zero-shot learning. In Medical Imaging with Deep Learning, pp. 402-415. PMLR, 2024.
Han, T., Xie, W., and Zisserman, A. Self-supervised co- training for video representation learning, 2021. URL https:/ /ttxxviivvvvv/////v//////////avl www./ogg///..oooooo abs/ 201 0. 09709.
Lin, Z., Yu, S., Kuang, Z.. Pathak, D., and Ramanan, Multimodality helps unimodality: Cross-modal few-shot learning with multimodal models, 2024. URL https: //arxiv. org/ abs 22301.0667...
He, K., Zhang, X., Ren, S., and Sun, J. Deep residual leamn ing for image recognition. In Proceedings of the IEEE conference on computer vision and paltern recognition, pp. 770-778, 2016.
06267.
Liu, C., Wan, Z., Cheng, S., Zhang, M., and Arcucci, R. Etp: Learning transferable ecg representations via ecg-text pre-training. In ICASSP 2024-2024 IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP), pp. 8230-8234. IEEE, 2024a.
He, P., Liu, X., Gao, J., and Chen, W. Deberta: Decoding- enhanced bert with disentangled attention. arXiv preprint arXiv:2006.0n) 03654, 2020.
Liu, C., Wan, Z.., Ouyang, C., Shah, A., Bai, W., and Ar- cucci, R. Zero-shot eeg classification with multimodal learning and test-time clinical knowledge enhancement. In Forty-first International Conference Machine Learn- ing. 2024b.
Hu, R., Chen, J., and Zhou, L. Spatiotemporal self- supervised representation learning from multi-lead eog signals. Biomedical Signal Processing and Control, 84: 104772,
Jin, Q., Kim, W., Chen, Q., Comeau, D. C., Yeganova, L., Wilbur, J., and Lu, Z. Medept: Contrastive pre-trained transformers with large-scale pubmed search logs for zero-shot biomedical information retrieval. Bioinformat- ics, 39(1 1:btad651, 2023.
Liu, F., Liu, C., Zhao, L., Zhang, X., Wu, X., Xu, X., Liu, Y., Ma, C.,, Wei, S., He, 7.,, al. An open evaluating the algorithms of clectrocardiogram rhythm and morphology abnormality detection. Journal of Med- ical Imaging and Health Informatics, 8(7):1368-1375, 2018.
database for
Kim, S., Kim, S., and Lee, J. Hybrid generative-comtrastivee representation learning, 2021. URL httth: org/ htrg/abs/en 611.60660000 62.
McKeen, K., Oliva, L., Masood, S., Toma, A., Rubin, B., and Wang. B. Ecg-fm: An open electrocardiogram foun- dation model. arXiv preprint arXiv:2408.0511 051788 2024.
Kiyasseh, D.,
Zhu, T., and Clifton, D. A. Clocs: Contrastive learning of cardiac signals across space, time, and pa- tients. In International Conference on Machine Learning, pp. 5606-5615. PMLR, 2021.
Na, Y., Park, M., Tae, Y., and Joo, S. Guiding masked representation learning to capture spatio-temporal rela- tionship of electrocardiogram. In International Confer- ence on Learning Representations, 2024. URL https: / //oppeeevil ew net. / forum? dot.oobbiddoooooooooooooo
Lalam, K., Kunderu, H. K., Ghosh, S., Kumar, H., Awasthi, S., Prasad, A., Lopez-Jimenez, F., Attia, Z. I., Asirvatham, S., Friedman, P., et al. Ecg representation learning with multi-modal ehr data. Transactions on Ma- chine Learning Research, 2023.
Oh, J., Chung, H., Kwon, J.-m., Hong, D.-g., and Choi, E. Lead-agnostic self-supervised learning for local and global representations of electrocardiogram. In Confer- ence on Health, Inference, and Learning, pp. 338-353, PMLR, 2022.
Li, G., Togo, R., Ogawa, T., and Haseyama, M. Self- knowledge distillation based self-supervised learning for covid-19 detection from chest x-ray images. In ICASSP 2022 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, May 2022a. doi: 10.1 10.10////casp44492.2222
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748-8763, PMLR, 2021.
9746540. URL http:// / dx doi. /ttg//...1099 ICASSP43922 2022. 9746540.
Li, J., Li, D., Xiong, C., and Hoi, S. Blip: Bootstrapping language-iman pre-training for unified languaee understanding and generation. International confer- ence on machine learning. pp. -29000 PMLR, 2022b.
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., and Liu, P. J. Explor- ing the limits of transfer learning with a unified text-to- text transformer, 2023. URL https: hatss///arxii //aaxiivvooo .org abs/1910.10683.
10

---

Boosting Masked ECG-Text Auto-Encoders as Discriminative Learners
Rasheed, H., Khattak, M. U., Maaz, M., Khan, S., and Khan, F. S. Fine-tuned clip models are efficient video
Yan, G., Liang, S., Zhang, Y., and Liu, F. Fusing trans- former model with temporal features for eeg heartbeat classification. In 2019 IEEE International Conference on Bioinformatics and Biomedicine (BIBM), pp. 898-905. IEEE, 2019.
learners, 2023. URL https: http/xxxivvoo/// orgoobgg//...ooooooo... / 2212. 03640.
Reyna, M. A., Sadr, N., Alday, E. A. P., Gu, A., Shah, J., Robichaux, C., Rad, A. B., Elola, A., Seyedi, S., Ansari, S., et al. Will two do? varying dimensions in electrocardiography: the physionet/compuuing cardiol- ogy challenge 2021. In 202/ Computing in Cardiology (CinC), volume 48, pp. 1-4. IEEE, 2021.
Yu, H., Guo, P., and Sano, A. Eeg semantic integrator (esi): A foundation ecg model pretrained with IIm-enhanced cardiological text. Transactions on Machine Learning Research (TMLR), 2024.
Zbontar, J., Jing, L., Misra, I., LeCun, Y., and Deny, S. Barlow twins: Self-supervised learning via redundancy reduction. In International conference on machine learn- ing, pp. 12310-12320. PMLR, 2021.
Ribeiro, A. H., Ribeiro, M. H., Paixão, G. M. M., Oliveira, D. M., Gomes, P. R., Canazart, J. A., Ferreira, M. P. S., Andersson, C. R., Macfarlane, P. W., Meira Jr., W., Schön, T. B., and Ribeiro, A. L. P. Automatic di- agnosis of the 12-lead ECG using deep neural net- work. Nature Communicatiions, 11(1):1760, 2020. doi: 15432-4.
Zhai, X., Mustafa, B., Kolesnikov, A., and Beyer, L. Sig- moid loss for language image pre- pre-training. In Proceed- ings of the IEEE/CVF International Conference Com- puter Vision, pp. 11975-11986, 2023.
http::/doi.orr/110. http://0...oo///10.0088844444400
Saeed, A., Ozcelebi, T., and Lukkien, J. Multi-task self- supervised learning for human activity detection. Pro- ceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, 3(2):1-30, 2019.
Zhang, H., Liu, W., Shi, J., Chang, S., Wang, H., He, J., and Huang, Q. Maefe: Masked autoencoders family of electrocardiogram for self-supervised pretraining and transfer learning. IEEE Transactions on Instrumentation and Measurement, 72:1-15, 2022a.
Siontis, K. C., Noseworthy, P. A., Attia, Z. I., and Friedman, P. A. Artificial intelligene--nhhaneed electrocardiography in cardiovascular disease management. Nature Reviews Cardiology, 18(7):465-478, 2021.
Zhang, W., Yang, L., Geng, S., and Hong, S. Self-supervised time series representation learning via cross reconstruc- tion transformer. IEEE Transactions on Neural Networks and Learning Systems, 2023.
Song, J., Jang, J.- Lee, B. T., Hong, D., Kwon, J.-m., and Jo. Y.-Y. Foundation models for ecg: Leveraging hybrid self-supervised learning for advanced cardiac diagnostics. arXiv preprint arXiv:2407.077 10, 2024.
Zhang, X., Zhao Z., Tsiligkaridis, T., and Zitnik, M. Self- supervised contrastive pre-training for time series via time- free-frey consistency. Advances in Neural Infor- mation Processing Systems, 35:3988-4000 4003, 2022b.
Tonekaboni, S., Eytan, D., and Goldenberg, A. Unsu- pervised representation learning for time series with temporal neighborhood coding, 2021. URL https: //arxiv.org org/ abs 221.. 00750.
Zhang, Y., Jiang, H., Miura, Y., Manning, C. D., and Lan- glotz, C. P. Contrastive learning of medical visual rep- resentations from paired images and text. In Machine Learning for Healthcare Conference, pp. 2-25. PMLR, 2022c.
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., and Polosukhin, I. Attention is all you need, 2023. URL https: http:///axvvvv.. org/ abs/ abbs/706..e 63762.
Zheng, J.. Guo, H., and Chu, H. A large scale 12-lead electrocardiogram database for arrhythmia study (version 1.0. O). PhysioNet 2022Available online httpphysionet orgcontentecg arrhythmial 10 Oaccessed on, 23, 2022.
Wagner, P., Strodthoff, N., Bousseljot, R.-D., Kreiseler, D., Lunze, F. I., Samek, W., and Schaeffter, T. Ptb-xl, a large publicly available electrocardiograhyy dataset. Scientific data, 7(1):1-15, 2020.
Wang, N., Feng, P., Ge, Z., Zhou, Y., Zhou, B., and Wang. 7. Adversarial spatiotemporal contrastive learning for clectrocardiogram signals. IEEE Transactions on Neural Networks and Learning Systems, 2023.
Xu, L., Lian, J., Zhao, W. X., Gong, M., Shou, L., Jiang, D., Xie, X., and Wen, J.-R. Negative sampling for con- trastive representation learning: A review. arXiv preprint arXiv:2206.000 002.000000222222 /2, 2022.

---

Boosting Masked ECG-Text Auto-Encoders as Discriminative Learners
A. Appendix
A.1. Data and Training Details.
We first visualize representative examples of ECG-text pairs from the MIMIC IV ECG dataset (Gow et al., 2023), as shown in Figure We also indicate the top 30 common unique reports (before merging) in Figure 5. Prominent terms such as "abnormal eog", "normal eeg", "atrial fibrillation", and "sinus tachycardia" indicate common diagnoses, which suggests prevalent cardiovascular conditions and typical annotations within this dataset.
(3) Atrall rbiillation. Possible anterior infarct- age undetermined intertor) ateral ST-T changes be due to myccar dial schama. Low QRS valage precondial leads. Abnornal ECG.
Probable accelerated Lefoward anis
()) Sinus arbhythmia. Normal ECG.
(9 Bradycardaa Normal ECG for rate.
[3 Snus fhythm. Abnomal A wove progresioo, early transtion.
(6) Strus rhythm. Left sais devistion 5000 with left anterior fascicula Mort. Abnormal EDG.
Pouitle atrial ventriculr Amar changes
bradytardia borderline block vollages precordia loacs.
4
Figure 4. Examples of ECG-text pairs in MIMIC IV ECG dataset (Gow et al., 2023). We visualize three leads II, V2) out of twelve.
Next, we provide more details on data configurations in Table 9, including data split, number of classes, metrics, and the corresponding tasks with the given downstream datasets. Particularly, the CODE-test dataset is from the work (Ribeiro al., 2020) with their model being trained on set of over 2 million ECG records from 1,676,384 different patients in 811 counties. We show our model's ' effectiveness by evaluating our performance on the same released test set of 827 samples, but in zero-shot manner. The samples are originally sampled at 400 Hz, with durations of either 10 seconds or seconds. Therefore, we resampled to 500 Hz and adjusted by truncating or padding with zeros as needed to get 10-second samples. For the gold standard (ground truth), two expert cardiologists provided their diagnoses. they agree with each other, their consensus becomes the gold standard. cases of disagreement, third specialist reviews their diagnoses and determines the final decision.
We also indicate important hyper-parameters during the fine-tuning process in Table 10. We keep training 200 epochs, batch size at 128, and learning rate at 0.001 for the first three datasets. When conducting full fine-tuning experiments, we only need to train 100 epochs and specifically lower the learning rates with 0,00005 and 0.0001 for Dx. and 1d. tasks, respectively.
Finally, for the distribution shift experiment, we follow the SCP-codes (classes) matching settings in (Liu et al., 2024b), as shown in Table 11. This is to support three dataset matches (PTBXL-Super and CPSC2018), (PTBXI-SSper and CSN), and
12

---

Boosting Masked ECG-Text Auto-Encoders as Discriminative Learners
possible inferior infarct age undete undetermined
normal ecg except for rate border line
inverime S sinus tachycardia,
ecg branch block inferior wav changes are nonspecific
romopecfft
right bundle
abnorma S
lateral I save changes are nonspecific
undetermined g bradycardia
rorst
probable
nereal vertent
conjuct
inferior
infarct
age
normal
ecgs
sinsin
ann Lire
atrial
fibrillation
Lim
low
myyte
qrs
ber
voltages in precordial leads
let
degres
possible anterior
infarct
age
undetermined
left Sundle brand
black
sihus
anterior
deviation left ventricular hypertr ophy poor
left t axis
le tward axiss
leads
prolonged qt interval
enteriee
wave progression probable normal variant
Figure 5. WordCloud visualization on the top 30 common unique reports from the MIMIC IV ECG dataset.
Table 9. Details on data configurations on five evaluated datasets. Here, LP, ZS are linear probing and zero-shot respectively, while FFT means full fine-tuning,
Dataset
Tasks
Metric
# Classes
Train
Valid
Test
PTBXI PTBXI--Super (Wagner et al., 2020)
LP,
AUC
17,084
2,146
2,158
PTBXL-Sub (Wagner et al., 2020)
LP, 7S
AUC
23
17,084
2,146
2,158
PTBXL-Form (Wagner et al., 2020)
LP, ZS
AUC
19
7,197
901
880
PTBXL-Rhythm (Wagner et al., 2020)
LP, ZS
AUC
12
16,832
2,100
2,098
CPSC2018 (Liu et al., 2018)
LP, ZS
AUC
4,950
551
1,376
CSN (Zheng et al., 2022)
LP, ZS
AUC
38
16,546
1,860
4,620
Physiome12021-DX. (Reyna et al., 2021)
FFT
CinC
26
32,640
4,079
4,079
Physionet2021-Id. (Reyna et al., 2021)
FFT
Accuracy
2,127
147,444
17,670
2,127
CODE-test (Ribeiro et al., 2020)
ZS
AUC
827
(CPSC2018 and CSN). It is worth noting that the None value indicates the target dataset does not have matching label for given labels in the source dataset.
A.2. Contrastive Learning Discussion.
Why Not Use ETM Only in Zero-shot Learning. As mentioned in the Method section, ETM functions as a contrastive learning technique in the masked auto-encoder architecture. However, it heavily relies on binary classification tasks with explicit ECG-text pairs to learn cross- correspondences. It is not designed for zero-shot learning which strongly requires the model to generalize unseen tasks classes without the need for such supervised pairings or fused information during training. This motivates to ETS boosting the model's zero-shot ability.
Why N3S Can Enhance The Performance. In medical datasets, particularly the MIMIC-IV ECG dataset (Gow et al., 2023), significant amount of duplicate highly similar text samples: among nearly 800,000 records, only approximately 1 180,000 are unique. For instance, over 100,000 samples share an identical text report, which is "sinus rhythm normal eeg". Randomly selecting negative samples for contrastive loss training is not suitable approach in this scenario. Therefore. we propose using the N3S technique to more effectively differentiate between similar and dissimilar samples, improving contrastive learning by selecting more meaningful negatives. Notably, during training, we observe that the ETM accuracy without N3S stagnates around 75%, while with N3S, exceeds 96%, demonstrating the significant impact of this approach.
13

---

Boosting Masked ECG-Text Auto-Encoders as Discriminative Learners
Table 10. Details on training configurations the fine-tuned datasets. For the optimizer, keep using Adam in all experiments.
Dataset
# Epoch
Batch size
Learning rate
PTBXL-Super (Wagner et al., 2020)
200
128
0,001
PTBXL-Sub (Wagner al., 2020)
200
128
0.001
PTBXL-Form (Wagner al., 2020)
200
128
0.001
PTBXI-Rhhyth (Wagner et al., 2020)
200
128
0.001
CPSC2018 (Liu et al., 2018)
200
128
0.001
CSN (Zheng et al., 2022)
200
128
0.001
Physionet202I-Dxx (Reyna et al., 2021)
100
256
0.00005
Physioner2021-Ic (Reyna et al., 2021)
100
256
0.0001
Table 11. Domain transfer category matching.
PTBXI-Super
CPSC2018
HYP
None
NORM
NORM
CD
IAVB, CRBBB, CLBBB
MI
None
STTC
STE, STD
PTBXI-SSuper
CSN
HYP
RVH, LVH
NORM
SR
CD
2AVB, 2AVB1, IAVB, AVB, LBBB, RBBB, STDD
MI
MI
STTC
STTC, STE, TWO, STTU, QTIE, TWC
CPSC2018
CSN
AFIB
AFIB
VPC
VPB
NORM
IAVB
LAVB
CRBBB
RBBB
STE
STE
PAC
APB
CLBBB
LBBB
STD
STE, STTC, STTU,
A.3. Enhancing Zero-shot Performance with LLMs.
(1) Response with merging subtypes reducing capability on new tasks
"AFIB":" Attiil Fibrillation, Paroxysmal Atrial Fibrillation, Persistent Fibrillation, Long-standing Persistent Atrial Fibrillation, Permanent Atrial Fibrillation."
"SEHYP": "septal hypertrophy, left ventricular septal hypertrophy, right ventricular septal hypertrophy, apical septal hypertrophy, mid-septal hypertrophy."
(2) Response showing limitations on LLM's searching and hallucination
"AF": "Atrial Flutter, Atrial Fibrillation, Paroxysmal Atrial
"BIGU": "Based on the input, generated the following subtypes and attributes for Bigeminal pattern ...Let me know if this meets your requirements!."
Flutter, Persistent Atrial Flutter, Long- standing Persistent Atrial Flutter."
Figure 6. Limitations on MERL's enhanced texts.
14

---

Boosting Masked ECG-Text Auto-Encoders as Discriminative Learners
Limitations in MERL's Approach. In zero-shot learning, models typically rely on category names alone to make predictions. However, by incorporating Large Language Models (LLMs)), can enhance the context by generating richer, clinically relevant descriptions of the categories, as discussed in MERL (Liu et al., 2024b). However, we observe two main drawbacks in their enhanced text reports, as shown in Figure 6: 1) MERL's performance heavily depends on their sub-types and attributes searching prompt and additional database. This leads to a limitation when testing detailed analysis with labels that are sub-types themselves. Moreover, this also raises suspicion about the performance when new tasks require labels that are not able to search sub-types and attributes in the database; 2) Following that point, MERL's enhanced texts might be uncontrollable in the outputs where the LLMs provide wrong sub-types or unnecessary context. For example, "Atrial Fibrillation" is already in "AFIB" type but is also shown to be in "AF" with other "Atrial Flutter" types from their settings.
LLMs in D-BETA We address the above points using straightforward prompt strategy with explicit instructions. Specifically, we employ prompt: "You are experienced cardiologist. For given clinical term such as 'normal ECG', your job is to describe each term clinically and apply your medical domain knowledge to include other relevant explanations that will help a text encoder like Flan-T5 fully understand medical concepts. Do not include any recommendations in the description." " This makes the LLM generate clinically accurate and more focused explainable descriptions, enhancing the text encoding without introducing irrelevant or redundant information. For example, with the code "AFIB", our prompt on GPT-4o can output: "Atrial Fibrillation (AFIB). Irregular and often rapid heart rate due to uncoordinated atrial activity.".
Additional Experiments. Here, we present additional experiments highlight the effectiveness of ETM loss and masking modeling techniques (e.g., MLM,, MEM). Specifically, we perform zero-shot classification with GPT-4o support (reported in AUC (%)) on four datasets: PTBXL-Super, PTBXL-Form, CSN, and CODE-Test.
Table 12. Impact of ETM. Results report zero-shot classification in AUC (%)
PTBXL-Super
PTBXL-Form
CSN
CODE-Test
w/ ETM
73.2
65.8
76.6
96.2
w ETM
76.2
66.1
76.3
96.8
As indicated in Table 12, the impact of ETM is demonstrated where, removing ETM slightly decreases performance across most datasets, particularly in PTBXL-Super (76.2 to 73.2). However, the effect on CSN is minimal, suggesting dataset-specific sensitivity to ETM.
Table 13. Impact of MLM and MEM. Results report zero-shot classification in AUC (%).
PTBXL-Super
PTBXL-Form
CSN
CODE-Test
w/o MLM + MEM 70.3
67.4
74.5
94.6
W MLM + MEM
76.2
66.1
76.3
96.8
Finally, we can observe that incorporating MLM and MEM noticeably improves performance across all evaluated datasets in Table 13. Especially, gains are observed in PTBXL- (+5.9%), and CODE-Test (+2.2%), demonstrating that the reconstruction tasks also play important role in enhancing the model's ability for better performance.
15


Reviews:
[{"review_id": "QTuseZw99r", "reviewer": "Reviewer_NfLP", "summary": "This paper proposes D-BETA, a novel contrastive masked transformer-based architecture to pre-train ECG signals and corresponding texts. The key components of the proposed approach include self-supervised learning for both ECG and medical texts, as well as fusion mechanism for both to enhance cross-learning. A nearest-neighbor negative sampling was also used to support contrastive learning. Experiment results and ablation studies were included.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their constructive feedback on our submission. We would like to address your comments below:\n\n> [R1-1]: “In some scenarios, the text info, especially from doctors, may be seen as a ground truth or label, rather than training data. Some discussion on this would be helpful, too.”  … “In evaluating and comparing the proposed method to other baselines, whether all other baselines incorporate the text information in training … if some of the other methods do not have the text info available, the comparison may not be a fair one.”\n\nFirstly, our model is not explicitly exposed to ground-truth labels during pre-training. **The textual reports in the MIMIC-IV-ECG dataset are machine-generated and serve as various descriptive inputs, not as discrete predefined label annotations**. \n\nFurthermore, additional text description can be beneficial, but importantly, we presented the efficient way to leverage them in an underexplored multimodal setting, optimizing performance in **the same downstream experiments** with the baselines: (1) Comparing with strong ECG-only SSL methods to show multimodal modeling power. (2) Comparing MERL (ICML publication) in the same multimodal aspect.\n\nMoreover, evaluated **downstream tasks such as fine-tuned classification and identification use unseen ECG recordings, no text is involved; or complete zero-shot tasks** (consistent with the latest close work like MERL). Therefore, we believe these comparisons are both fair and meaningful. \n\n\n> [R1-2]: “It would be helpful to expand the literature review here, as this is the key contribution of this paper.”\n\n**We acknowledge that discussing more details of recent related ECG-text works would be more helpful to our work. Therefore, we kindly synthesize more recent works available, and by this, we also want to highlight the rapidly growing developments and attention to this medical domain**:\n\nFirstly, [1] pretrains on a large private dataset using a standard contrastive SSL with ResNet-like ECG and BERT-based text encoders. Their evaluation is also limited to a few downstream tasks and diseases. [2] build on a similar contrastive modeling design but introduces prompt-based zero-shot inference. While insightful, their evaluation remains relatively narrow. ESI [3] opens another interesting angle when using a RAG pipeline to produce auto-generated detailed report data for the pretraining stage using multiple datasets (including MIMIC IV, PTB-XL, and Chapman). However, the contrastive modeling approach is still relatively similar (Bert-like text encoder, fully convolutional Convnext model, no MEM, ETM, or lead augmentation), and evaluation is relatively modest. \n\nRecently, as ECG representation learning continues to evolve, a few works have begun to explore more practical, user-end applications. For example, [4,5] leverage instructed multimodal LLMs to deal with ECG report generation and clinical question answering. These methods largely benefit from a well-pretrained ECG encoder, which D-BETA also can be potentially adapted.\n\n[1] Lalam, Sravan Kumar, et al. \"Ecg representation learning with multi-modal ehr data.\" TMLR (2023).\n\n[2] Liu, Che, et al. \"Etp: Learning transferable ecg representations via ecg-text pre-training.\" ICASSP (2024).\n\n[3] Yu, Han , et al. \"Ecg semantic integrator (esi): A foundation ecg model pretrained with llm-enhanced cardiological text.\" (2024).\n\n[4] Zhao, Yubao, et al. \"ECG-Chat: A Large ECG-Language Model for Cardiac Disease Diagnosis.\" (2024).\n\n[5] Yang, Kai, et al. \"ECG-LM: Understanding Electrocardiogram with a Large Language Model.\" HDS (2025).\n\n\n> [R1-3]: “Please explain when applying MAE to the ECG (multi-channel), how are the different channels masked”\n\nAs described in Section 3.1 (Lines 139-143), we apply random lead masking, where entire ECG channels are independently masked with a probability of 0.5. This lead-wise masking encourages the model to learn robust representations across varying input configurations.\n\n\n> [R1-4]: “Please explain explicitly the formation of the latent representation z as a fusion of the outputs from the ECG-specific and text-specific encoders. Overall, the paper would benefit from some theoretical analysis/heuristic on how cross-learning and fusion improve model performance.”\n\nOur fusion module is designed as stacked cross-attention blocks, which allows interaction between multi-lead ECG signals and semantic text embeddings (Lines 175-180) before decoding. This produces essential joint embeddings for our learning objectives: for example, the ETM task requires a unified representation to determine whether an ECG-text pair is matched, which naturally depends on information from both modalities. Similarly, MLM and MEM benefit as one modality provides useful context to reconstruct the other (e.g., ECG noise or text ambiguity compensation). This fusion also facilitates future downstream tasks such as ECG report generation, where signal-based text generation is critical.", "claims_and_evidence": "Generally well supported claims.", "methods_and_evaluation_criteria": "Evaluation criteria are generally sound. One comment/question is that in evaluating and comparing the proposed method to other baselines, whether all other baselines incorporates the text information in training. The text information itself is rich and sometimes contain even more information than the ECG itself, so if some of the other methods do not have the text info available, the comparison may not be a fair one.", "theoretical_claims": "NA", "experimental_designs_or_analyses": "Experimental designs are generally sound. It would be helpful to also discuss the computational efficiency of the proposed approach, when comparing to similar approaches that can incorporate both ECG and texts.", "relation_to_broader_scientific_literature": "In discussing literature related to using both ECG and text in self-supervised or contrastive learning, the only paper cited was MERL. It would be helpful to expand the literature review here, as this is the key contribution of this paper.", "essential_references_not_discussed": "NA", "other_strengths_and_weaknesses": "None", "other_comments_or_suggestions": "Please explain when applying MAE to the ECG (multi-channel), how are the different channels masked (randomly, same time window masked simultaneously across channels, etc.)\n\nPlease explain explicitly the formation of the latent representation $z$ as a fusion of the outputs from the ECG-specific and text-specific encoders. This seems to be the crucial step to enhance cross-learning of the two components.\n\nOverall, the paper would benefit from some theoretical analysis/heuristic on how cross-learning and fusion of the ECG and text component improve model performance. \n\nIn some scenarios, the text info, especially from doctors, may be seen as a ground truth or label, rather than training data. Some discussion on this would be helpful, too.", "questions_for_authors": "NA", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {"ethical_review_concerns": "None"}}, {"review_id": "LnN2SQNA3f", "reviewer": "Reviewer_9VgN", "summary": "This paper introduces the D-BETA framework for joint pre-training of ECG signals and their corresponding clinical text reports, aiming to learn cross-modal self-supervised representations. The method integrates generative tasks—specifically, masked language modeling (MLM) and masked ECG reconstruction (MEM)—with a discriminative task based on ECG-text matching via contrastive learning. Additionally, the paper proposes the ETS loss to enhance the model’s discriminative capability and employs a nearest-neighbor negative sampling strategy (N3S) to effectively select negative samples. Extensive experiments on several public datasets (e.g., PhysioNet 2021, PTB-XL, CSN, CPSC2018, and CODE-test) demonstrate that D-BETA achieves significant performance improvements under full fine-tuning, linear probing (even when only 1% of the training data is used), and zero-shot scenarios. Ablation studies further validate the contributions of key components such as the ETS loss, N3S strategy, and the Flan-T5 text encoder.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> [R2-1]: \"The overall task design seems to largely build upon existing multimodal joint modeling methods and appears more like an aggregation of several tasks, lacking sufficient novelty.\"\n\nWhile our framework builds upon several established tasks, to the best of our knowledge, **we are the first to design and investigate the contrastive masked autoencoder ECG-text pre-training**. As novelty is subjective, we focus on the contribution and significance of our work to the ICML (especially with the primary area of Applications->Health/Medicine) community:\n\n - We propose D-BETA that specially uses a transformer-based ECG encoder and the Flan-T5 model (overlooked in ECG-clinical community), together with attention-based fusion modules and decoders.\n - We propose and investigate the insight of discriminative ETS loss into masked ECG-Text autoencoder implementations (with ETM, MLM, MEM and arbitrary lead augmentation), enabling robust multimodal representation learning (unexplored in the literature). \n - We are the first to introduce the N3S technique to address data unexplored redundancy in the MIMIC-IV ECG dataset, improving the quality of negative samples and boosting model performance. \n - We conduct extensive experiments across zero-shot, linear probing, and fully fine-tuned settings, demonstrating that our approach consistently outperforms strong baselines across more than 100 cardiac conditions.\n\nWe also verify the effectiveness of proposed components by conducting the diverse ablation studies and necessary appendix, as as acknowledged by the reviewers NfLP (\"Experimental designs are generally sound\"), ffzJ (\"Experiments are thorough and show clear improvement over existing state-of-the-art\") and TuyF (“providing a thorough analysis of its capabilities in practical applications”). Finally, we emphasize the practical implication of the ECG research field and sincerely believe that our work reflects a principled design and empirical effort rather than simple aggregation.\n\n> [R2-2]: \"Although the paper cites a substantial number of related works, it might benefit from discussing more recent advances in multimodal fusion or self-supervised learning in the context of ECG data.\"\n\nThank you for the suggestion. This is closely related to R1-2 in which we discuss more recent works regarding multimodal ECG-Text fusion. \n\n\n> [R2-3]: \"The paper does not include an experiment comparing ETS directly against a conventional softmax-based loss … can the authors further demonstrate the efficiency and superiority of the ETS loss over traditional methods?\"\n\nThank you for the thoughtful point. Our ETS loss is inspired by the Sigmoid contrastive loss proposed in SigLIP[1], which has already **demonstrated strong theoretical and empirical advantages over softmax-based losses. These “Sigmoid” benefits remain efficiency and superiority in our implementation since we kindly introduce the unaffected ETS heads with Dense layer and Tanh activation while simply setting t=1 and b=0 (as we design positive-negative balanced batches during training)**. Building on this foundation, we focus our main ablations on showing ETS’s benefit in our specific multimodal ECG-text setup.  \n\n[1] Zhai, Xiaohua, et al. \"Sigmoid loss for language image pre-training.\" Proceedings of the IEEE/CVF international conference on computer vision. 2023.\n\n> [R2-4]: \"In the loss module, the authors emphasize the efficiency advantages of the proposed approach. Could the authors elaborate on how the N3S strategy impacts the overall computational burden? Does it affect the model’s efficiency and scalability?\"\n\nWe appreciate this point and acknowledge the importance of discussing the computational aspect of using N3S. First, the FAISS index is constructed once before training using precomputed text embeddings in the small Flan-T5 space. During training, we only perform efficient nearest-neighbor retrieval using this index and this action will introduce some tradoff. Importantly, FAISS is known for its impressive speed and scalability in large-scale vector search tasks, and in our case, produces comparable runtime overhead and still doesn’t affect the efficiency and scalability much. **Empirically, on 1× NVIDIA A100-40GB, the one-time loading of the model and FAISS index takes ~1.2 seconds while the average (over 1000 samples) time using FAISS was approximately 0.00219 seconds (compared to 0.00002 seconds without it)**.\n\n> [R2-5]: \"Have the authors reproduced and fairly compared related methods under a unified experimental setting to ensure a fair comparison of performance?\"\n\nRegarding baseline comparison, we used the results reported in the original baseline papers. In D-BETA, we always aim for the fair comparisons by strictly following the baselines’ released data splits, preprocessing and downstream configurations. \n\n---\n\nWe sincerely appreciate your valuable feedback and have responded to your reviews and hopefully these meet your expectations and acknowledgement.", "claims_and_evidence": "Claims:\nThe authors claim that combining the advantages of generative and discriminative learning can significantly improve the alignment and representation of cross-modal features from ECG signals and text reports, thereby enhancing downstream tasks such as cardiac disease diagnosis and zero-shot inference.\n\nEvidence:\nThe paper provides extensive experimental results, reporting superior performance (e.g., improvements in AUC, and notable gains in fine-tuning and linear probing scenarios) compared to state-of-the-art methods. In addition, ablation experiments corroborate the positive impact of the ETS loss and the N3S strategy on the overall model performance.", "methods_and_evaluation_criteria": "Methods:\nThe paper employs a Transformer-based ECG encoder and a pre-trained Flan-T5 text encoder, using a cross-attention module to fuse features from both modalities. Three task branches are set up corresponding to masked language modeling (MLM), masked ECG reconstruction (MEM), and ECG-text matching (ETM).\n\nEvaluation Criteria:\nThe experiments are conducted on multiple datasets under full fine-tuning, linear probing, and zero-shot settings, with evaluation metrics mainly including classification accuracy and AUC. Overall, the chosen methods and evaluation criteria effectively capture the characteristics of the problem and the capabilities of the proposed model.", "theoretical_claims": "Fusion of Generative and Discriminative Learning:\nThe authors argue that combining generative tasks (e.g., MLM and MEM) with a discriminative task (ECG-text matching) in a unified self-supervised framework can complement each task’s strengths. While the generative tasks help capture the fine-grained structure of the data, the discriminative task reinforces the separation of cross-modal features, leading to more robust and discriminative representations.\n\nSigmoid-based ETS Loss:\nThe paper introduces a novel ETS loss that computes the matching probability for each ECG-text pair independently using the Sigmoid function and optimizes the distance between positive and negative pairs via binary cross-entropy loss. The authors contend that this approach is more efficient than traditional softmax-based contrastive losses—which require global normalization—and thus provides a lightweight, memory-friendly alternative.\n\nRationale Behind the N3S Strategy:\nThe authors provide theoretical justification for using the nearest-neighbor negative sampling (N3S) strategy instead of random negative sampling. By selecting negative samples that are semantically distant from the positive samples in the pre-trained text embedding space, the strategy effectively enhances the contrastive learning process. This claim is based on the recognition of inherent semantic similarities in medical text data and underscores the importance of semantic divergence in negative sampling.", "experimental_designs_or_analyses": "The experimental design is comprehensive, covering full fine-tuning, low-resource linear probing (using 1% of training data), and zero-shot settings, and validating the model across multiple public ECG datasets. Ablation experiments assess the contributions of critical components (e.g., various text encoders, different numbers of Transformer layers, and the presence or absence of the ETS loss and N3S strategy).\n\nHowever, it is noted that the experimental results for zero-shot ECG classification are consistent with those reported in “Zero-Shot ECG Classification with Multimodal Learning and Test-time Clinical Knowledge Enhancement” and “Lead-agnostic Self-supervised Learning for Local and Global Representations of Electrocardiogram.” It remains unclear whether the authors have reproduced these works under identical experimental conditions for a fair comparison.\n\nMoreover, the paper does not directly compare the ETS loss against a traditional softmax-based loss; instead, it only shows the effect of removing ETS. This approach demonstrates the contribution of the discriminative task but does not conclusively prove the superiority of the sigmoid-based ETS loss over conventional softmax formulations.", "relation_to_broader_scientific_literature": "This work is closely related to previous self-supervised ECG learning methods (e.g., CMSC, 3KG, ST-MEM) and multimodal fusion approaches (e.g., MERL), while also drawing inspiration from visual-language pre-training frameworks like CLIP. The innovative incorporation of the N3S strategy using FAISS for efficient vector retrieval and the integration of generative and discriminative tasks (e.g., ETS loss) provide a novel perspective on cross-modal feature learning.", "essential_references_not_discussed": "Although the paper cites a substantial number of related works, it might benefit from discussing more recent advances in multimodal fusion or self-supervised learning in the context of ECG data to further enrich the literature background and offer a more comprehensive comparison.", "other_strengths_and_weaknesses": "Strengths:\n\n- The paper innovatively combines generative and discriminative self-supervised learning.\n- The proposed method demonstrates particularly strong performance in low-resource and zero-shot settings, showing promising potential for practical applications.\n\nWeaknesses:\n\n- The overall task design seems to largely build upon existing multimodal joint modeling methods and appears more like an aggregation of several tasks, lacking sufficient novelty.\n- Regarding the ETS loss, while the use of a Sigmoid function to avoid the computational burden of global normalization is an interesting idea, the experimental validation and theoretical justification for its effectiveness in enhancing discriminative power remain insufficient.", "other_comments_or_suggestions": "none", "questions_for_authors": "The paper does not include an experiment comparing ETS directly against a conventional softmax-based loss—only an ablation where ETS is removed is provided. How can the authors further demonstrate the efficiency and superiority of the ETS loss over traditional methods?\n\nIn the loss module, the authors emphasize the efficiency advantages of the proposed approach. Could the authors elaborate on how the N3S strategy impacts the overall computational burden? Does it affect the model’s efficiency and scalability?\n\nHave the authors reproduced and fairly compared related methods under a unified experimental setting to ensure a fair comparison of performance?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "c23CYQucdF", "reviewer": "Reviewer_ffzJ", "summary": "This paper presents a self-supervised pretraining method for jointly learning from electrocardiograms (ECGs) and text. Their method, D-BETA, combines modality-specific masked modeling, a sigmoid matching loss, and a nearest neighbor negative sampling strategy to enhance performance. Results demonstrate superiority over state-of-the-art ECG pretraining strategies on a variety of downstream datasets and tasks.\n\n## Update after rebuttal\nI have read the authors' review and will maintain my original recommendation. I thank the authors for clarifying my questions/concerns, but the rebuttal does not change my overall stance on the submission.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> [R3-1]: \"Were baselines pretrained from scratch on the same data as D-BETA, were their model weights taken as is and used for fine-tuning, or were results taken directly from their respective papers?\"\n\nRegarding baseline comparison, we used the results reported in the original baseline papers. They can use **different datasets for the pre-training stage but we keep the fair comparisons by strictly following the baselines’ released data splits, preprocessing and downstream configurations**. \n\n> [R3-2]: \"What is the authors’ justification for using a language model pretrained on natural text (that is then fine-tuned)? Could the approach potentially be improved by using a text encoder pretrained on cardiovascular reports specifically, perhaps negating the need to fine-tune the text encoder?\"\n\nThank you for your thoughtful question. **Flan-T5 is pre-trained on large data of natural language that possibly captures rich medical context from domain-specific website (e.g., cardiovascular disease related forums)**, which indicates it is helpful. Additionally, trained with huge corpus, Flan-T5 shows **strong adaptability across unseen tasks. If further fine-tuning Flan-T5 in D-BETA, it could perform better on specific cardiovascular reports**. As shown in Table 7, we can see that Flan-T5 after fine-tuned during D-BETA pre-training stage is nearly 1% better than fine-tuned Med-CPT, a biomedical-pretrained model used in MERL (as a SOTA approach). That said, we agree that using a Flan-T5 text encoder first pre-trained specifically on cardiovascular reports might also offer additional benefits.\n\n> [R3-3]: \"Can the authors provide an ablation study on the negative sampling? I see a discussion of why N3S might be helpful in Section A.2, but do not see numerical results backing this up.\"\n\nWe kindly note that the **effectiveness of N3S is already reported in Table 6**, where removing N3S causes a 2% drop in zero-shot performance. Additionally, in Appendix A.2, we report that the ETM accuracy without N3S stagnates at ~75%, while with N3S it exceeds 96%. We believe these results support N3S’s impact of semantically-aware negative sampling. \n\n> [R3-4]: \"Do the authors intend to release code and model weights? This will be important to ensure reproducibility.\"\n\n**Yes, as noted in the Impact Statement part (Lines 442-451), we will publicly release the pretrained models and code upon acceptance. We are happily committed to ensuring reproducibility and enabling future research**.\n\n> [R3-5]: \"Technical novelty is limited (though the additions are clearly helpful): the sigmoid loss is borrowed from SigLIP – per the authors’ admission - and the combination of contrastive and generative approaches has been seen in C-MELT [1].\"\n\nWe respectfully understand the reviewer’s perspective and would like to refer to R2-3 and R2-1 for further discussions on our related justifications and contributions. **We do not directly follow the presentation in their paper but make suitable adjustments in our modeling context and our core contributions go beyond using ETS loss**.  We kindly note that [1] is not discussed here due to the sensitive policy of the conference. \n\n> [R3-6]: “A few additional citations to ECG-text foundation models could be included [2-4]. I wouldn’t say these are “essential” in that their omission/inclusion changes my stance on the paper, but it is helpful to inform the reader that this is a growing space with many relevant approaches.”\n\nThank you for your kind suggestions. [2] is a comprehensive review paper that synthesizes various aspects of ECG foundation including background, existing datasets, common modeling approaches, and various practical applications. [3] also focuses on the ECG foundation model but proposes to use LLMs to enrich ECG report text with medical knowledge then perform a signal-text-label contrastive learning. Meanwhile, [4] does not pretrain ECG-text directly but interestingly views ECG signals as a language, with QRS complexes as words and rhythms as sentences. **Alongside the discussed works in R1-2, we believe these reflect the growing momentum in this space as you kindly noted**. \n\n---\n\nWe appreciate your time to review our work. At this end, we have addressed your comments and we would be grateful if you are satisfied with our responses and could acknowledge our rebuttal.", "claims_and_evidence": "Yes, claims appear to be sound and supported by evidence.", "methods_and_evaluation_criteria": "Yes. This study uses large-scale, standard ECG datasets and evaluation metrics that are consistent with prior work.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "Experimental design appears sound. I am curious how baseline results were derived, however. Were baselines pretrained from scratch on the same data as D-BETA, were their model weights taken as is and used for fine-tuning, or were results taken directly from their respective papers?", "relation_to_broader_scientific_literature": "This study represents an addition to the growing collection of vision-language models for multimodal ECG-text representation learning. This method features a few additional existing previously existing techniques to boost performance beyond current published state-of-the-art.", "essential_references_not_discussed": "C-MELT [1] is another ECG-text pretraining method that combines contrastive and generative objectives; this is an important omission to me.\n\nA few additional citations to ECG-text foundation models could be included [2-4]. I wouldn’t say these are “essential” in that their omission/inclusion changes my stance on the paper, but it is helpful to inform the reader that this is a growing space with many relevant approaches.\n\n[1] C-MELT: Contrastive Enhanced Masked Auto-Encoders for ECG-Language Pre-Training.\" arXiv preprint arXiv:2410.02131 (2024).\n\n[2] Han, Yu, et al. \"Foundation Models in Electrocardiogram: A Review.\" arXiv preprint arXiv:2410.19877 (2024).\n\n[3] Tian, Yuanyuan, et al. \"Foundation model of ECG diagnosis: Diagnostics and explanations of any form and rhythm on ECG.\" Cell Reports Medicine 5.12 (2024).\n\n[4] Jin, Jiarui, et al. \"Reading your heart: Learning ecg words and sentences via pre-training ecg language model.\" arXiv preprint arXiv:2502.10707 (2025).", "other_strengths_and_weaknesses": "*Strengths*:\n- The paper is generally well-written with clear presentation\n- Experiments are thorough and show clear improvement over existing state-of-the-art\n\n*Weaknesses*:\n- Some methodological details surrounding baseline implementation could be clarified\n- Technical novelty is limited (though the additions are clearly helpful): the sigmoid loss is borrowed from SigLIP – per the authors’ admission – and the combination of contrastive and generative approaches has been seen in C-MELT [1].", "other_comments_or_suggestions": "- L11 on RHS: Change “e.g.” -> “e.g.,”\n- L86: Remove extra comma after “D-BETA”\n- L374 on RHS: Remove extra space before footnote in “ECG examples ”", "questions_for_authors": "1.\tWere baselines pretrained from scratch on the same data as D-BETA, were their model weights taken as is and used for fine-tuning, or were results taken directly from their respective papers? \n2.\tWhat is the authors’ justification for using a language model pretrained on natural text (that is then fine-tuned)? Could the approach potentially be improved by using a text encoder pretrained on cardiovascular reports specifically, perhaps negating the need to fine-tune the text encoder?\n3.\tCan the authors provide an ablation study on the negative sampling? I see a discussion of why N3S might be helpful in Section A.2, but do not see numerical results backing this up.\n4. Do the authors intend to release code and model weights? This will be important to ensure reproducibility.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "OfUn7FmLjh", "reviewer": "Reviewer_TuyF", "summary": "This paper introduces D-BETA, a cross-modal pre-training framework designed for self-supervised learning of ECG signals and textual reports. D-BETA combines the strengths of generative and contrastive learning by leveraging masked language modeling and masked ECG reconstruction to recover missing data. Additionally, it employs contrastive loss functions alongside a nearest-neighbor negative sampling strategy to improve alignment between the two modalities. Extensive experiments conducted on multiple public datasets demonstrate that D-BETA significantly outperforms existing methods, particularly in downstream tasks such as zero-shot learning and linear probing. These findings underscore its strong generalization capabilities and highlight its potential for advancing diagnostic applications.", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> [R4-1]: Lack of originality  / Limited originality: The framework combines multiple loss functions (ETS, ETM, MLM, MEM), none of which are original contributions by the authors. Furthermore, the coefficients used to balance these losses are not systematically explored or justified. This makes the work appear more like an engineering effort rather than a novel machine learning contribution, suggesting it may be better suited for a clinical application journal rather than ICML … The novelty of the contributions raises questions about the framework's suitability for a high-impact venue like ICML.\n\nWe appreciate your feedback and would like to kindly refer to the key contributions of our works as in R2-1. While the individual loss components are inspired by prior works, our contribution lies in how they are specifically adapted, integrated for the underexplored masked ECG-text multimodal autoencoder setting. We acknowledge that we did not systematically explore loss weighting coefficients, which can be a valuable direction for future work. However, **given the context of investigating ECG-text multimodal pre-training by proposing a novel generalized contrastive masked autoencoder framework that has already demonstrated to surpass existing benchmarks, we kindly believe your conclusion about “an engineering effort” only based on the lack loss coefficients may overlook our approach**. \n\n**We also emphasize the growing importance of ECG-based modeling in both machine learning and clinical applications. As you pointed, “ECG analysis is an important task for medical application”, ECGs are non-invasive, widely used physiological signals, and clinical reports provide contextual information that complements the waveform, but the combination between them remains underexplored in dynamic SSLs. Therefore, we hopefully aim for ICML (Primary Area: Applications->Health / Medicine) as a suitable venue to foster this impactful direction.**\n\n> [R4-2]: Heavy reliance on CLIP-style ETS loss and Flan-T5: In Table 6, removing the ETS loss (which essentially reduces the model without using vanilla CLIP loss) results in a significant drop in performance. This indicates that the model heavily depends on the CLIP loss, while other components contribute relatively little. Similarly, replacing T5 with Med-CPT in Table 7 also leads to a performance decline. These observations raise concerns about whether the improvements stem primarily from leveraging more powerful off-the-shelf text encoders rather than from the proposed framework itself.\n\nWe would like to respectfully correct a potential misinterpretation in your examples. First, in our attempt to conduct ablation studies, it is ordinary when our proposed components have better performance effects than when not using them. **In Table 6, while removing the ETS loss leads to a performance drop, removing other components (e.g., Flan-T5 or N3S) also results in comparable degradation. In Table 7, Flan-T5 is the encoder we proposed in our framework, and is slightly better than Med-CPT (the best in MERL). We can also see that even with a weaker encoder (e.g., BERT), the remaining architecture still outperforms the last row in Table 6 (same BERT usage)**. Therefore, we sincerely believe the improvements reflect the synergistic design of the overall framework, rather than over-dependence on a specific component.\n\n> [R4-3]: Inconsistent ablation studies on individual losses\nIn Appendix Tables 12 and 13, the authors ablate ETM and MLM losses but fail to provide a consistent and systematic comparison of each individual loss component. This lack of clarity makes it difficult to discern which parts of the framework actually contribute to performance improvements. Additionally, the absence of detailed ablation studies undermines the ability to assess the incremental value of each loss function.\n\n**We kindly note that our Table 12 evaluates the effect of ETM, while Table 13 evaluates the joint effect of MLM and MEM purposely, which together form our overall generative reconstruction objective.**\n\nFurthermore, rather than the need of detailed individual loss effect, we believe that focusing more on our main ablation studies which examine the incorporation of core components are more important in our framework, with the effects of N3S, ETS, Flan-T5 (Table 6), variations of text encoders (Table 7), the scalability in the ECG encoder (Table 8).\n\n---\n\nWe hope these clarifications address your concerns. We thank you for your review and hope that you will consider raising our score, as we believe our work offers valuable contributions to the ICML community.", "claims_and_evidence": "yes", "methods_and_evaluation_criteria": "yes", "theoretical_claims": "this is not theory work", "experimental_designs_or_analyses": "Yes, the authors implement comprehensive experiments.", "relation_to_broader_scientific_literature": "ECG analysis is a important task for medical application[1].\n[1] AI-enabled electrocardiography alert intervention and all-cause mortality: a pragmatic randomized clinical trial. Nature Med", "essential_references_not_discussed": "No", "other_strengths_and_weaknesses": "#### **Strengths**\n- **Comprehensive Downstream Task Evaluation**: The authors evaluate their model's performance across a wide range of downstream tasks, providing a thorough analysis of its capabilities in practical applications.\n\n#### **Weaknesses**\n1. **Lack of Originality in Loss Design**  \n   - The framework combines multiple loss functions (ETS, ETM, MLM, MEM), none of which are original contributions by the authors. Furthermore, the coefficients used to balance these losses are not systematically explored or justified. This makes the work appear more like an engineering effort rather than a novel machine learning contribution, suggesting it may be better suited for a clinical application journal rather than ICML.\n\n2. **Heavy Reliance on CLIP Loss**  \n   - In Table 6, removing the ETS loss (which essentially reduces the model without using vanilla CLIP loss) results in a significant drop in performance. This indicates that the model heavily depends on the CLIP loss, while other components contribute relatively little. Similarly, replacing T5 with Med-CPT in Table 7 also leads to a performance decline. These observations raise concerns about whether the improvements stem primarily from leveraging more powerful off-the-shelf text encoders rather than from the proposed framework itself.\n\n3. **Inconsistent Ablation Studies**  \n   - In Appendix Tables 12 and 13, the authors ablate ETM and MLM losses but fail to provide a consistent and systematic comparison of each individual loss component. This lack of clarity makes it difficult to discern which parts of the framework actually contribute to performance improvements. Additionally, the absence of detailed ablation studies undermines the ability to assess the incremental value of each loss function.\n\n4. **Limited Originality**  \n   - The work lacks significant novelty, as all the loss functions are borrowed from prior studies, and the authors merely combine them without introducing substantial innovation. This diminishes the originality of the contribution and raises questions about the framework's suitability for a high-impact venue like ICML.", "other_comments_or_suggestions": "please see the weakness.", "questions_for_authors": "please see the weakness.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: We thank the authors for their submission.  The study is interesting, well written, and contains a thorough empirical evaluation across a variety of ECG datasets and tasks.  Reviewers shared the concern that the technical approach contains limited novelty.  However, the main contribution is focused on the applied ECG domain, and though there is limited novel ML, the succinctly presented and thorough ablation study conducted show that this set of components appears to meaningful imbue ECG representations with valuable information.  As reviewer ffzJ puts it, "Experiments are thorough and show clear improvement over existing state-of-the-art."  During the rebuttal period, authors clarified points to valid questions raised in the initial round of reviews.  The authors are encouraged to incorporate those clarifying points into their manuscript.
Decision: Accept (poster)