From Token to Rhythm: A Multi-Scale Approach for ECG-Language Pretraining
Authors: Fuying Wang
Jiacheng Xu
Lequan Yu
Venue: ICML 2025 poster (2025)
Keywords: ECG
Representation Learning
Multimodal Pretraining
Primary area: applications->health_medicine
TL;DR: We proposes an ECG-Language pretraining model leveraging multi-scale cross-modal supervision.

Abstract: Electrocardiograms (ECGs) play a vital role in monitoring cardiac health and diagnosing heart diseases. However, traditional deep learning approaches for ECG analysis rely heavily on large-scale manual annotations, which are both time-consuming and resource-intensive to obtain. To overcome this limitation, self-supervised learning (SSL) has emerged as a promising alternative, enabling the extraction of robust ECG representations that can be efficiently transferred to various downstream tasks. While previous studies have explored SSL for ECG pretraining and multi-modal ECG-language alignment, they often fail to capture the multi-scale nature of ECG signals. As a result, these methods struggle to learn generalized representations due to their inability to model the hierarchical structure of ECG data. To address this gap, we introduce MELP, a novel Multi-scale ECG-Language Pretraining (MELP) model that fully leverages hierarchical supervision from ECG-text pairs. MELP first pretrains a cardiology-specific language model to enhance its understanding of clinical text. It then applies three levels of cross-modal supervision—at the token, beat, and rhythm levels—to align ECG signals with textual reports, capturing structured information across different time scales. We evaluate MELP on three public ECG datasets across multiple tasks, including zero-shot ECG classification, linear probing, and transfer learning. Experimental results demonstrate that MELP outperforms existing SSL methods, underscoring its effectiveness and adaptability across diverse clinical applications. Our code is available at https://github.com/HKU-MedAI/MELP.

From Token to Rhythm: A Multi-Scale Approach for ECG-Language Pretraining
Fuying Wang" I Jiacheng Xu 1 Lequan Yu
Abstract
Electrocardiograas (ECGs) play vital role in monitoring cardiac health and diagnosing heart diseases. However, traditional deep learning ap- proaches for ECG analysis rely heavily on large- scale manual annotations, which are both time- consuming and resource-intensive to obtain. To overcome this limitation, self-super learn- ing (SSL) has emerged as promising alternative, enabling the extraction of robust ECG represen- tations that can be efficiently transferred to var- ious downstream tasks. While previous studies have explored SSL for ECG pretraining and multi- modal ECG-language alignment, they often fail to capture the multi-scale nature of ECG signals. As a result, these methods struggle to learn gen- eralized representations due to their inability model the hierarchical structure of ECG data. To address this gap, we introduce MELP novel Multi-scale ECG-Language Pretraining (MELP) model that fully leverages hierarchical supervi- sion from ECG-text pairs. MELP first pretrains cardiology-ppecifie language model to enhance its understanding of clinical text. It then applies three levels of cross-modal supervision at the token, beat, and rhythm levels-to to align ECG signals with textual reports, capturing structured information across different time scales. We eval- uate MELP on three public ECG datasets across multiple tasks, including zero-shot ECG classifi- cation, linear probing, and transfer learning. Ex- perimental results demonstrate that MELP out- performs existing SSL methods, underscoring its effectiveness and adaptability across diverse clini- cal applications. Our code is available at htt / //github. com/ HKU MedAI /MELP.
Rhythm View
Beat View
Q
Token View
W
Figure 1. Illustration the multi-scale view ECG signals. Rhythm Level: Captures the full ECG recording, reflecting the heart's global electrical activity over time. Beats Level: Segments each rhythm into discrete heartbeat tokens, isolating individual cardiac cycles for localized analysis. Token Level: Further de- composes each heartbeat into finer-grained temporal components, enabling granular feature extraction.
1. Introduction
Electrocardiograms (ECGs) are widely used for monitoring cardiac health and diagnosing cardiovascular diseases. The standard 12-lead ECG records electrical activity from differ- ent perspectives, capturing both temporal and spatial charac- teristics of the heart's function. Advances in deep learning have significantly improved the analysis of these signals, enhancing the analysis of their underlying patterns (Yan et al., 2019; Ebrahimi et al., 2020; Siontis al., 2021).
Self-supervised learning (SSL) has emerged as promising solution for ECG analysis, enabling meaningful representa- tion learning without the need for labeled data. Existing SSL approaches for ECGs rely on either contrastive (Kiyasseh et al., 2021; Oh al., 2022: McKeen et al., 2024) or gen- erative methods (Na et al., 2024: Hu et al., 2023: Jin et al., 2025), but most focus solely ECG signals without lever- aging complementary clinical knowledge. With the growing availability of clinical documentation. multi-modal learn- ing specifiifl ECG-language alignment- gained attention. Recent studies (Zhao et al., 2024; Yu et al., 2024; Pham et al., 2024) have explored methods for link- ing ECG signals with textual interpretations. Notably, Liu
'Equal contribution School of Computing and Data Science, The University Hong Kong, Hong Kong SAR, China. Corre- spondence to: Lequan Yu <lqyu@hku.hk>. hku.hk>>
Proceedings of the ndnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

From Token to Rhythm: A Multi-Scale Approach for ECG-Language Pretraining
al. (2024a) introduced MERL, a zero-shot ECG-language framework that uses contrastive learning to align ECG recordings with clinical reports. Despite these advances, existing models primarily focus on global ECG-to-text align- ment, overlooking the rich, multi-scale structure of ECG signals (Figure 1). For example, the thythm-level view cap- tures overall temporal rhythm of the ECG, the beat-level view analyzes the characteristics of each cardiac cycle, and token-level view interprets specific waveform segments. All of these views are crucial for comprehensive ECG interpre- tation. To address this limitation, a more detailed approach, incorporating token-, beal-, and rhythm-level representa- tions, is essential for capturing comprehensive ECG features critical for precise diagnostics.
2. Related Works
2.1. ECG Representation Learning
Self-supervised learning (SSL) has demonstrated significant efficacy in leveraging unlabeled data across diverse domains, including natural language processing (Devlin, 2018; Dong et al., 2019; He et al., 2020b), computer vision (Wu et al., 2018; Grill et al., 2020; Caron et al., 2021), and time-series analysis (Yue et al.. 2022; Nie et al., 2022: Eldele et al., 2021). Recently, SSL has been extended to electrocardio- gram (ECG) signal analysis, enabling robust representation learning for downstream pathology detection tasks (Baevski et al., 2020; Gopal et al., 2021). Existing methodologies in ECG SSL primarily fall into two categories: contrastive and generative approaches.
To address this gap, we propose MELP, novel Multi-scale ECG-Language Pretraining model that fully exploits the multi-scale structure of ECGs at the token, beat, and rhythm levels, incorporating detailed cross-modal knowledge from clinical text. A key component of MELP is a dedicated pretraining stage, where we first train a cardiology-specinn language model before jointly training on paired ECG-text datasets. This step enhances the model's ability to inter- pret medical terminology and align textual information with ECG signals. To capture fine-grained modality interactions at the token level, we introduce an ECG captioning task, enabling the model to generate descriptive representations of short waveform segments. At the beat level, we extract heartbeat embeddings from token representations and sen- tence embeddings from word tokens, applying a contrastive learning objective align beats with their corresponding clinical descriptions. Finally, at the rhythm level, we in- corporate a global contrastive loss to learn robust represen- tations of full ECG recordings. This multi-scale approach bridges the gap between raw ECG signals and their clinical interpretations, allowing MELP to learn highly transferable representations applicable across a range of tasks. We eval- uate MELP on three public ECG datasets, demonstrating its superiority over existing self-supervised and multi-mode models in zero-shot classification, linear probing, and cross- institutional transfer learning.
Contrastive methods (Sangha et al., 2024), currently the dominant paradigm, aim to maximize similarity between representations of augmented views of the same instance (positive pairs) while minimizing similarity with unrelated instances (negative pairs). Widely adopted frameworks such as SimCLR (Chen et al., 2020) and MoCo (He et al., 2020a) have inspired ECG-specific adaptations. For exam- ple, Kiyasseh et al. (Kiyasseh et al., 2021) employ tailored signal augmentations (e.g.. lead masking, noise) to generate positive ECG pairs for contrastive training.
Generative approaches, such as ST-MEM (Na et al., 2024) and HeartLang (Jin al., 2025), learn representations by reconstructing masked portions of ECG signals. These meth- ods (Yu et al., 2023) typically occlude temporal segments (e.g., P-waves, T-waves) or entire beats and train models to recover the original waveform. HeartLang further intro- duces a tokenization strategy that segments ECG recordings by detecting QRS complexes, mapping these physiological events to discrete embeddings via a trainable codebook.
Recent efforts (Oh et al., 2022; McKeen et al., 2024; Song et al., 2024) combine contrastive and generative objectives to develop ECG foundation models. These frameworks are pre-trained on heterogeneous ECG datasets to learn generalizable representations. While promising, existing methods operate solely on uni-modal ECG data, overlooking the rich semantic correlations between ECG signals and clinical text reports, which is limitation our work explicitly addresses.
In summary, our contributions are threefold:
A novel Multi-scale ECG-Language Pretraining model (MELP) that hierarchically integrates clinical text knowledge for improved ECG representation learning.
2.2. ECG-Language Pretraining
A structured pretraining framework with explicit cross- modal supervision at three clinically meaningful levels: token, beat, and rhythm.
Multi-moda representation learning aims to integrate infor- mation from diverse modalities by aligning their respective representations. Notable prior works, such as CLIP (Rad- ford et al., 2021), employ contrastive methods to align im- age and text data. However, relatively few studies (Li et al., 2024; Han et al., 2024; Tian et al., 2024) have ex- plored multi-modal representation learning in the context
Comprehensive evaluation on three public ECG datasets, achieving of-te-of-the-nn performacee in zero- shot classification, linear probing, and cross-domain transfer learning.
2

---

From Token to Rhythm: A Multi-Scale Approach for ECG-Language Pretraining
of ECG data analysis. For instance, ESI (Yu et al., 2024) leverages Large Language Model (LLM) to reinterpret electronic health record (EHR) data and guide ECG rep- resentation learning, while MERL (Liu et al., 2024a) pro- poses a framework that aligns ECG signals with clinical text reports to achieve strong zero-shot classification per- formance. Additional efforts, such as C-MELT (Pham et al., 2024), adopt generative approaches for both modali- combinigg Masked Language Modeling (MLM) with Masked ECG Modeling (MEM)-ane and incorporate cross- modal contrastive learning for ECG-text alignment. Simi- larly, ECG-Chat (Zhao et al., 2024), inspired by the CoCa architecture (Yu et al., 2022) (a state-of-the-anl contrastive framework), aligns ECG recordings with textual reports.
3.3. Multimodal Pretraining
Motivation. Cardiologists interpret ECG signals in a hierar- chical manner, analyzing features at multiple scales from individual waveform components (tokens) heartbeats (beats) and overall rhythm. This hicrarchical approach forms the basis for many clinical diagnostic criteria. We observe that all three levels are essential for accurate ECG in- terpretation, as illustrated by the following examples (More examples are provided in Appendix B):
Token-level: In diagnosing Atrial Fibrillation, clini- cians look for absence of P waves and QRS duration usually <120ms. These features are localized within short waveform segments, corresponding to the token level. (LB, 2011)
Despite these advances, existing models predominantly fo- on deriving global representation from the whole ECG recordings, overlooking fine-grained local patterns. Conse- quently, they do not effectively capture task- task-specific features at localized intervals, potentially limiting their adaptability to diverse downstream applications.
For sinus rhythm, each QRS complex should be preceded by a normal P wave True. This cri- teria requires to analyze two waveform components to- gether across the entire heart beat. (Mattu et al., 2019)
Beat-level:
Rhythm-level: Diagnosing Left Anterior Fascicular Block (LAFB) involves identifying left axis devia- tion-e.g. negative deflections in leads II, III, and aVF, and positive deflections in leads I and aVL (Mattu et al., 2019). This assessment depends on recognizing patterns over the entire ECG lead.
3. Method
3.1. Overview
Figure 2 provides overview of MELP which learns gener- alized ECG representations through multi-scale cross-modal supervision in a self-supervised manner. We begin by in- troducing cardiology language pretraining strategies (Sec- tion 3.2) to equip the text encoders with domain- specific knowledge. Next, we present our multimodal pretraining framework in Section 3.3, which leverages fine-grained cross-modal supervision to align ECG and text modalities effectively. Finally, we describe the evaluation protocol for transferring the pretrained framework to downstream tasks in Section 3.4.
Inspired by this structured approach, our model, MELP, integrates three levels of cross-modal supervision for ECG- language pretraining. Formally, we define the ECG encoder as f(=:0) and text encoder g(; Φ). where the ECG encoder is random initialized and the text encoder is initialized from the pretrained language model (Sec. 3.2). Given an ECG sample X and its paired ECG report T, we omit sample index (i) for clarity.
3.2. Cardiology Language Pretraining
Token-view: Learning to Generate Captions. To leverage the benefits of multimodal generative pretraining, we adopt an encoder- architecture for ECG-language learn- ing. The text decoder is designed to generate ECG reports with fine-grained detail, predicting tokenized text autore- gressively based on ECG token embeddings. For each ECG sample X, our ECG encoder f produces token-level ECG embedding E E LLDDD, where L, is the number of ECG tokens and the feature dimension. To summarize these embeddings, we use an attention pooler (Yu et al., 2022) with 128 learnable query tokens, resulting in É E 12 12 The attention pooler consists of a single multi-head atten- tion layer, where the encoder output serves as both keys and values. Following a GPT-style autoregressive framework. the text decoder maximizes the conditional likelihood of the paired report through next-word prediction. For a token sequence T (< BOS >, W1, WN; < EOS >), the text
As witnessed in the prior work (Boecking et al., 2022) in medical vision-language pretraining, the language plays a significant role in learning generalized visual representa- tions. To maximize the language model's utility for cardiol- ogy, we pretrain a text encoder using a cardiology-foctt focused corpus, building on the MED-CPT query encoder (Jin et al., 2023). The corpus is curated from three sources: cardiology- related data from PubMed', Wikipedia", and the MIMIC-IV- ECG training set (Gow et al., 2023), following the approach of HeartBERT (Gwon et al., 2024). We use the masked language modeling objective (Devlin, 2018) to pretrain the model. The details of pretraining the cardiology language model is presented in Appendix A.
https///wwwncbi nlm.nih, gov/hooee/deevlllllapp// hhtips:/en wikipedia. wrw./wi////iiiiiieiiiiia
3

---

From Token to Rhythm: A Multi-Scale Approach for ECG-Language Pretraining
Encoding
Token-level Supervision
Rhythm-Levee Supervision
Beat-Level Supervision
L2-Lead EOGs
eneader
Transformer ook
Globa locS represemaaions
beats lenel represertations
Gross attwentons
Sentence based attention weighted ECO representations
Projectto mulimadall space
Transformer
Text: tokens
Takeniter
<bowv
sirus
Fransdor mI Disoddrr
Attention Pooler
Lonal Biat Cont rasioo Lous
Gaball Contrattive los
frythhm
Caption
EGG bears sentences ciilarity mathis
Text Reports
add Ipfo
agppe
Learnable querr for beath
multiform vontricutor
"shus rhyehm.
Project nubmodda SpHOR
Senterce Tegresentation
Slbbil Rott apprssanattrn
prematurre compleses. and loft."
Text encoder
Sentencee Aggregation
Figure 2. Overview of MELP: MELP incorporates three levels of supervision token, beat, and rhythm to guide ECG- ECG-language pretraining. At the token level, fine- ane-grained ECG representations serve as queries for a transformer decoder, which reconstructs paired ECG reports using captioning loss (CLM). At the beat level, token-level ECG features aggregated into beat-level representations via an attention pooling layer, while text representations are grouped into sentence-level embeddings. A similarity matrix between ECG beats and text sentences is used reweight these embeddings, optimizing beat-level contrastive loss (Croca)). At the rhythm level, beat-level ECG fcatures and sentence-level text embeddings are further aggregated through average pooling to generate global representations, which are optimized using global contrastive loss (Cg).
decoder optimizes:
latent space using attention pooler. Specifically, we intro- duce 10 Iearnable tokens to hierarchically summarize beats within a 10-second ECG segment. While we use a default setting of 10 tokens in the main manuscript, an empirical analysis presented in Appendix C.2 shows that this is not necessarily the optimal choice. We leave the exploration of more adaptive and clinically informed heartbeat alignment strategies for future work.
N
LLM(S)= = - logp( wy wo:
(1)
where [ denotes the learnable parameters of the text de- coder. The text decoder is randomly initialized and trained with rracher-forcing (Williams & Zipser, 1989) to enhance computational efficiency and learning speed. An additional advantage of this generative pretraining approach is its flex- ibility in adapting to downstream tasks, such as ECG report generation and ECG Question Answering (ECG-QA).
Clinical observations suggest that transient abnormal beats often correspond to specific sentences in ECG reports. To capture this fine- grained alignment, we propose a beat- sentence matching mechanism. Let B E R Nnx represent beat embeddings obtained from the ECG encoder's attention- pooled outputs, and let S E SxD denote sentence embed- dings, where each sentence embedding is computed by av- eraging its word tokens. A projection layer-either a linear transformation or a two-layer MI.P- both embeddings into a shared latent space using functions PE (for ECG) and (for Text). For simplicity, we continue to denote the resulting embeddings as B and S. For each sentence embed- ding S(I) RR,, we compute an iieettion-eiighee. beat embedding (() using this equation:
Discussion. While some ECG reports provide high-level rhythm summaries (e.g.. "sinus rhythm")) many also include detailed descriptions of waveform-level abnormalities. Ex- amples of such observations are provided in Appendix B. Besides, high-level findings, like "sinus rhythm",, still de- pend on lowelevel indicators such as P wave consistency and PR interval regularity. By providing full waveform features to the decoder, token-level pretraining allow the model to learn these relationships and generate reports with varying levels of granularity. It may encourage the model analyze local features and implicitly learn these indicators.
No
Beat view: Heart Beat-Sentence Alignment. ECG signals can be segmented into individual heart beats, allowing for more interpretable cardiological analysis (Jin et al., 2025). However, strictly relying on R- R-peak detection may disrupt inter-beat morphological information and temporal relation- ships. To mitigate this, we aggregate beat embeddings in the
B(1) = Σ SnS(()
(2)
1=1
where the attention weight on is defined as:
exp( B())/)
a(l)
(3)
Σ Nu ()))))

---

From Token to Rhythm: A Multi-Scale Approach for ECG-Language Pretraining
Table J. Details the number of samples in cach split for each downstream dataset.
#. Samples
MIMIC-IV-ECG
PTBXL-Sub Rhyxt
PTBXL-Form
PTBXI-Super
CPSC2018
CSN
Train
745,447
16,832
17,084
7,197
17,084
4,950
16,546
Validation 15,171
2,100
2,146
901
2,146
1,860
Test
2,098
2,158
880
2,158
1,376
4,620
Here (.) denotes cosine similarity, T1 = 0.25 is tempera- ture hyperparameter. The similarity between an ECG-text pair is then computed by aggregating similarities between attention- attention-weighhed beat embeddings and the corresponding text embedding:
where A and A control the contributions of the language modeling and local alignment losses, respectively. Based on empirical results, we set \ 2 and X = 0.2 for pretrain- ing. The experimental results can be found in Sec. 4.3.5.
3.4. Transferring into Downstream Tasks
g
[ T) = log expl exp((B(),,50))/z
(4)
For zero-shot evaluation, directly use the global embed- dings X (ECG) and TÇ (Text) for retrieval- tasks.
where 72 = 0.1 is another temperature hyperparameter.
For fine-tuning, we first extract beat-level embeddings be- fore the projection layer pg- These embeddings are then aggregated via average pooling to obtain global feature vector. linear classification layer is added on top to gener- ate per-class predictions.
To optimize alignment, we define the local contrastive loss for a minibatch of size B as:
B Σ
exp (x,
CLocal
log
(5)
B i=1
1= Ix,)
In the linear probing setup, we freeze the entire network and train only the linear layer.
where is the sample index.
The total local loss is then given by: Loccal (C ( This loss function ensures robust heartbeat- sentence alignment, preserving both local and global ECG- text relationships.
4. Experiments
4.1. Experimental Setup
4.1.1. PRETRAINING TASK
Rhythm view: ECG-Report Alignment. Inspired by the success of CLIP (Radford et al., 2021), contrastive loss has proven effective in learning transferable multimodal representations. Following this principle, we introduce an instance-level contrastive loss to establish high-level cross- modal supervision between ECG signals and text reports.
Pretraining Dataset. For the pretraining stage, we uti- lize the MIMIC-IV-ECG vl.0 database (Gow et al., 2023), comprising 800,035 ECG recordings from 161,352 unique patients. Each recording consists of a 10-second waveform sampled at 500 Hz. The database provides multimodal alignment through clinical text reports, with up to 18 textual reports paired with each ECG recording. We adopt prepro- cessing protocols adapted from (Liu et al., 2024a), including text normalization (e.g., lowercase conversion, punctuation removal, and special character elimination). To ensure data quality, we exclude: (1) ECG recordings containing empty or NaN (Not-a-Number) values, (2) text reports with fewer than four tokens, and (3) ECG samples lacking paired tex- tual annotations. This rigorous curation process results in a final dataset of 760,618 high-quality ECG-text pairs.
To obtain global representations, we compute the ECG em- bedding by averaging all beat embeddings, while the text embedding is represented by the [CLS] token. For the i-th ECG-text pair, we denote these global embeddings as X (ECG) and T (text). Their similarity is computed using cosine similarity as (X/, ) The global alignment loss Lg 1 (CCg +
+ te) The fg- is defined following the
InfoNCE formulation (Oord et al., 2018):
exp( (xp( T///)) Σe (xp,
Ce-t = B log
(6)
Implementation Details. The ECG encoder is based on the Wav2Vec 2.0 architecture (Baevski et al., 2020), which integrates multi-layer convolutional neural network (CNN) feature extractor with transformer encoder. The ablation results justifying our selection of the Wav2Vec 2.0 architec- ture as the ECG encoder are included in Section 4.3.3. We use the AdamW W optimizer with an initial learning rate of 2e-4, a weight decay of 0.2, and a cosine annealing learning
where 7 is learnable temperature hyperparameter, and B denotes the batch size.
Overall Pretraining. Our framework is trained by jointly optimizing three loss functions:
fog + A1 LIM + X2 # Lrocal
5

---

From Token to Rhythm: A Multi-Scale Approach for ECG-Language Pretraining
Table 2. Linear probing performance (AUC [%] of MELP and baseline models across multiple datasets. Results are reported for different training data proportions (1%, 10%, and 100%) The best and second- becond-best results are highlighted in bold and underlined. respectively.
Methous
PTRNI -Rbythm
PTRXI-Form
-Super
CIPC77018
Training Dne
100%
SimCLR (Chen 2000)
51.41
05.27
75.54
BYOL (Grll et al. 2020)
43.99
74.40
77.17
57.16
67.44
71.64
61.63
70.82
71.20
73.83
74.42
78.75
BarkowTwins (Zhentarctal. 2021)
50.12
73.54
77.62
70.84
60.99
65. 14
22:87
75.96
78.41
72.70 75
76.3 99
60.72
77.43
MuCCaa3 (Class ct al. 2021)
51.38
71.66
74.33
55.88
69.21
76.69
50.32
63.71
71.31
73.19
76.65
73.
62.13
76.35 74
75.29
54.61
74.36
77.68
SimSiam (Chen & He. 20011
69.30
69.47
75.92
62.52
09.3 51
76, 50
55.16
62.91
71.31
75.15
72.00
15.63
58.35
72.89
75.31
58.25
58..1
77.41
TS-TCC
al. 2021)
43.34
69.48
78.23
53.54
65.98
77.57
48,04
61.79
71.15
70.75
75.58
73.91
57.07
73.62
76.72
55.26
58.48
76.79
CLOCS (Kayasseh 2011)
47.19
57.96
72.55
76.24
57.79
72.65
08.91
73.36
76.31
77.78
77.59
54.38
76.13
CMSC RLM (Oh 2022)
76.24
69.20
90.71
85.01
81.15
64.88
65.53
92.61
65.65
ASTCL (Wang 2004)
57.59
72.51
77.31
79.51
(7hang et al., 30239
47.44
73.52
74.41
98
70.82
78.67
59.40
08.73
69.60
78.24
77.34
58.01
76.4%
87:03
56.21
73.70
78,30
EXGGUM (McKeen et 2024)
$1.81
91.39
92.70
13.25
$1.91
16.07
00.95
74.99
85.54
78.67
$4.80
66.47
52.18
99.52
93.26
21.51
51,12
58,59
ST-MEM (No et sl., 20244
51.12
65.44
74.85
34.12
57.86
63.59
53.71
39.99
65.07
61.12
66.87
71.36
56.60
63.32
70.39
59.77
66.87
71.36
HeanLang (Jia et al. 2025)
02.00
76.22
90.34
64.68
79.34
88.91
58.70
63.99
80.23
78.94
53.59
67.52
60.44
66.26
77.87
57.94
58.93
52.49
MFRI (Iin et 2004e)
53.33
$2.88
66.34
64
90
80.56
84.72
58.26
72.45
79.65
82.39
26.22
88.67
70.33
85.5 92
90.57
66.60
62.74
57.05
MRI P. (Durs)
88.83
94.88
96.91
79.22
84.40
83.46
63.41
76.71
33.30
85.82
K7.61
87.87
88.54
91.75
94.32
78.25
64.83
90.17
Table 3. Zero-shot classification performance (AUC [%) of MELP and baseline models across multiple datasets.
Methods
CSN
PTBXL-Form PTBXL-Rhythm
PTBXL- PTB
CPSC2018
Average
MERL (Liu al., 2024a)
74.4
78.5
65.9
75.7
74.2
82.8
75.3
MELP (Ours)
77.6
85,4
69.1
81.2
76.2
84,2
79.0
Gains
+3.2
+6.9
+3.2 +5.5
+2.0
+1.4
+3.7
rate scheduler. MELP is pretrained for 100 epochs with per-device batch size of 64. Training is stopped early if the zero-shot prediction performance on the validation sets does not improve for five consecutive epochs. Please refer to our code for more details. All experiments are conducted on four NVIDIA GTX 3090 GPUs.
Implementation Details. For zero-shot evaluation, we use prompts driven by knowledge of GPT-4, following the approach in (Liu et al., 2024a). Linear probing tasks ad- here strictly to the predefined train-validation-tes splits from (Liu et al., 2024a). We conducted linear probing using 1% 10% and 100% of the training data for each task follow- ing (Liu et al., 2024a). All downstream tasks evaluated using AUROC (Area Under the Curve). We use batch size of 128 and train for 50 epochs, with early stopping similarly triggered based on the validation AUC. Further implementation details are provided in Appendix E.
4.1.2. DOWNSTREAM TASKS
Downstream Datasets. We evaluate our pre-trained MELP across three publicly available benchmarks: PTB-XL (Wag- ner et al., 2020), CSN (Zheng et al., 2022), CPSC2018 8 (Liu et al., 2018). A summary of dataset statistics is presented in Table 1, with additional details in Appendix D. Key charac- teristics of these datasets are summarized below:
4.2. Quantitative Results
1.2.1. EVALUATION ON LINEAR PROBING FOR ECG CLASSIFICATION
PTB-XL comprises 21,837 12-lead ECG recordings from 18,885 patients, each sampled at 500 Hz with a 10-second duration and annotated with cardiac diagnostic labels. Fol- lowing the methodology of MERL (Liu et al., 2024a), we stratify the dataset into four subgroups (super, sub, form, and rhythm) for granular evaluation. Training, validation, and test splits adhere to the protocol established by ( (Wagner et al., 2020).
Table 2 presents the linear probing results comparing MELP with baseline methods. We evaluate our MELP against both unimodal self-supervised approaches, including TS- TCC (Eldele al., 2021), CLOCS (Kiyasseh et al., 2021), ASTCL (Wang et al., 2024), CRT (Zhang et al., 2023), ST-MEM (Na et al., 2024), and HeartLang (Jin et al., 2025), as well as multimodal self-supervised method such as MERL (Liu et al., 2024a). MELP consistently improves classification performance across six tasks, achieving the highest accuracy in 16 out of 18 evaluation settings and ranking second in the remaining two. Its advantage is partic- ularly evident when using only 1% of training data, where it surpasses the second-best method in AUROC by mar- gins of +7.3%% +5.98% +2.46%, +3.43%, +6.36% and +6. 6..7%% respectively. These results underscore MELP's effectivenesss in low-data scenarios, highlighting its poten- tial for real-world applications where labeled medical data
CPSC2018 This resource contains 6,877 standard 12-lead ECG recordings sampled at 500 Hz, annotated with 9 cate- gorical labels. We replicate the experimental configuration of MERL. (Liu et al., 2024a) for consistency.
CSN contains 23,026 samples recorded 500 Hz over 10- second intervals, this dataset includes 38 distinct diagnos- tic labels. For downstream evaluation, we adopt the train- validation-test partitioning scheme proposed by MERL (Liu et al., 2024a).
6

---

From Token to Rhythm: A Multi-Scale Approach for ECG-Language Pretraining
Table 4. Performance under data distribution shift. "Source Domain" refers the dataset used for linear probing with the frozen pre-trained ECG encoder, while "Target Domain" represents the corresponding test set. The Best and Second best results are shown in Bold and underlined.
Source Domain
PTBXL-Super CSN
CPSC2018
CSN
Zero shot
Training Ratio
Average
Target Domain
CPSC2018
PIBXL-Super
CSN
PIBXL-Super
CPSC2018
SimCLR (Chen et al., 2020)
X
100%
69.62
73.05
56.65
66.36
59.74
62.11
65,22
BYOL- et al., 2020)
X
100%
70.27
74.01
57.32
67.56
60.39
63.24
65.63
BarlowTwing (Zhonlar el al., 2021)
X
100%
68.98
72.85
55.97
65.89
58.76
61.35
64.13
MoCo-v3 (Chen et ol., 2021)
X
100%
69.41
73,29
56,54
66. 12
59.82
62.07
64.21
SimSiam (Chen & He, 2021)
X
100%
70.06
73.92
57.21
67.48
60.23
63.09
65.31
TS-TCC (Eldelc al., 2021)
X
100%
71.32
75.16
58.47
68.34
61.55
64.48
66.55
CLOCS (Kiyasseh al., 2021)
X
100%
68.79
72.64
55.86
65.73
58.69
61.27
63.83
ASTCL (Wang et al., 2024)
X
100%
69.23
73.18
56.61
66.27
59.74
62.12
64.19
CRT (Zhang et al., 2023)
X
100%
70.15
74.08
$7.39
67.62
60.48
63.33
65.51
ST-MEM (Na et al., 2024)
X
100%
76.12
84.5
62.27
75.19
73.05
64.66
72.63
MERL (Liu ot al., 2024a)
0%
88,21
78.01
76.77
76.56
74.15
82.86
79.42
MELP (Ours)
0%
87.75
74.11
77.89
80.32
74.67
82.72
79.58
Table 5. Ablation results of loss functions on 6 linear probing tasks. The first row indicates training with only the instance-leve contrastive loss Cg. The Best and Second-best results are shown in Bold and underlined.
4.
Gis Gronal
1%
PTBXI--Rbp
10%
100%
1%
PTBXI--Fom
10%
100%
15
PTBXI--uhh
10%
100%
1%
PTAXI-SSper
10%
100%
1%
CPSC22018
10%
100%
1%
CSN
10%:
100%
Average
K3.78
57.93
72.14
82.07
77.32
81.977
94.36
84.55
87.24
87.52
75.52
57.07
92:57
75.94
$2.04
82.51
77.64
79.44
85.21
52.95
63,80
76.91
71.41
76.67
82.97
78.75
82.80
85.18
64.19
73.05
85
69.81
79.37
54.41
76.10
81.01
59.88
96.67
49.81
67.82
81.41
66.14
81.38
84.76
79.94
87.49
67.73
64.18
84.08
93.17
55.89
79.77
88.79
78.89
83:25
89.87
94.86
56.58
12.71
81,99
79.61
82.14
85.84
87.18
87.56
83:74
92.77
74.36
50.48
57.11
92.92
54 36
88.44
05.20
57.22
72.07
82.95
81.20
82.80
85.42
84,90
87.25
87.57
76.97
86.31
92.
73.37
81.43
81 $1.50
82.32
85.83
94.65
96,91
63.41
76.71
83.30
79.22
84,40
87.46
85.82
K7.61
87.87
88.54
91.75
94.32
78.25
84.83
90.17
85.78
is scarce. With full training data (100%), MELP contin- ues to rank first or second for all datasets, improving AUROC by +4.21%, +1.06%, and +1.28% on PTBXL- Rhythm, CPSC2018, and CSN, respectively. This demon- strates MELP's ability achieve superior performance even with abundant labeled data, setting strong upper bound for ECG classification.
space. For non non-zero-shot self supervised learning (SSL) baselines, we apply linear probing using 100% of the source domain training data. For zero-shot models (MERL), we follow the target-source category matching protocol out- lined in Appendix E. This evaluation measures MELP's ability to generalize across diverse clinical settings. As shown in Table 4, MELP achieves the highest performance in three out of six settings and ranks second in two. No- tably, it also achieves the best average performance across all six tasks. Specifically, our MELP outperforms MERL by + 1111,, +3.76% +0.52% on the CPSC2018 ++ PTXBL.- Super, CPSC2018 - CSN and CSN - PTXBL--Sper set- tings, respectively. These results highlights MELP's robust- ness and its ability to handle distribution shifts effectively.
4.2.2. EVALUATION ON ZERO-SHOT ECG CLASSIFICATION
We evaluate MELP's zero-shot classification performance against the multimodal baseline MERL (Liu et al., 2024a). As shown in Table 3, MELP achieves sfate-of-the-arr re- sults across all tasks, demonstrating significant performance improvements. Specifically, MELP demonstrates improve- ments of +3.2%, +6.9% +3.2%, +5.5%, +2.0%, +1.4% on CSN, PTBXL-Rhythm. PTBXL-Form, PTBXL-Sub, PTBXL--Suer, and CPSC2018 respectively. Moreover, MELP achieves an average performance gain of 3.7% over the MERL. These findings highlight MELP's effectiveness in zero-shot ECG classification, reinforeing its potential for clinical applications where evaluation on diverse down- stream tasks is required without additional training.
4.3. Analysis of Our Framework
4.3.1. ABLATION ON MULTI-SCALE SUPERVISION
To evaluate the impact of multi-scale supervision, includ- ing token-level supervision (LLM). beat-level supervision (LTocat), and rhythm-level supervision (Cg), we conduct ablation studies over different variants. This allows us to assess each supervision level's contribution. As shown in Table 5, The first three rows present models trained with single isolated supervision (one loss each). Rows four and five correspond to variants without token-level super- vision (LLM) and without beat-level supervision (Llocal), respectively. The final row shows the performance of the full model. Our full model achieves the highest perfor-
4.2.3. EVALUATION ON TRANSFER LEARNING FOR ECG CLASSIFICATION
To assess robustness to domain shifts, we evaluate transfer learning performance using test datasets that differ in dis- tribution from the pretraining data but share the same label
7

---

From Token to Rhythm: A Multi-Scale Approach for ECG-Language Pretraining
Table 6. Ablation results of loss functions on 6 zero-shot classification tasks.
CLM Looo
-TB PTBXI--Form -Ryhi
PTBXI--Super
CPSC2018
CSN
Average
79.6
68.7
77.1
77.2
83.9
75.0
76.9
84.1
67.4
74.0
74.1
81.9
76.8
76.4
82.8
64.4
79.0
75.9
81.1
76.9
76.7
85.4
69.1
81.2
76.2
84.2
77.6
79.0
Table 7. Ablation results of ECG encoders. We have used RLM as the augmentation technique for ECG by default. CMSC can't 't easily integrate into our model since it needs to split the ECG into two parts and performs contrastive learning.
BDG enender
PTBXL Rhythm
10%
1001s
PTBNL Form
10%
100%
PTBXL-Sub
10%
10055
PTBXL Super
1055
10015
CPSC2018
105
1001
CSN
10%
100%
Average
1eshet-18
85.10
90.11
94.31
62.82
73.59
79.23
75.59
81.72
85.70
85.84
K6.99
57.24
83.31
89.75
93.35
68.79
82.12
89.71
83.07
Waw 2.0
88.83
94.65
96.91
63.41
76.71
83.30
79.23
84,40
87.46
85.82
87.61
87.87
88.54
91.75
94.32
78.25
$4.83
90.17
85.78
Wav2Ver CMSC
58.25
94.82
75.55
82.57
82:29
87.52
55.10
71.89
$1.00
82.97
Table
Ablation results of pretrained comain-specific language model on 6 linear probing tasks.
PTRXI RKhythm
PTHXI_-Fom
PTBXI-Sub
PTRXI -Spper
CPSC2018
CSN
Text
1%
10%
100%
1%
10%
100%
17
10%
100%
1%%
10%
100%
1%6
10%
100%
156
10%
100%
Average
88.61
94.95
96.72
75.34
82.94
84.17
87.77
91.33
75.82
83.95
89.51
85.43
88.83
91.65
96.91
63.41
76.71
83.30
79.22
84.40
87.46
85.82
87.61
87.87
88.54
91.75
94.32
78.25
84.83
90.17
85.78
and BERTScore to assess the lexical and semantic qual- ity of the generated reports. As shown in Table 9, MELP significantly outperforms PULSE, demonstrating strong ca- pabilities in fine-graine ane-grained ECG understanding and report generation. These results highlight the potential of MELP to support clinically relevant diagnostic tasks that require detailed, fine-grained predictions. Additionally, we evaluate the model on patient identification, with results reported in Table 13. A more comprehensive analysis of fine-grained prediction performance remains an important direction for future research.
Table 9. ECG report generation metries ([%D) on 500 curated samples from PTB-XL report dataset.
Models
Size
BLEU
BLEU-
METEOR
ROUGE
BERTScore Fl
FULSE (Lis et 2024b)
7B
5.12
0.83
13.76
5.15
10.95
MRLPP
364M
13,02
1.77
11:28
18.90
44J08
mance on most benchmarks, outperforming the best partial- supervision variant by an average AUROC gain of 2.86%. Furthermore, we observe that variants without rhythm- level supervision (Cg) perform significantly worse, while all configurations including Lg achieve AUROCs above 82%. This demonstrates that global-level modti- align- ment is essential for effective ECG representation learning. This empirically demonstrates the importance of hierarchi- cal oupervision-inneeraatnng rhythm-level (Cg), beat-level (Loocal), and token-level (LLM) signals for learning clin- ically meaningful ECG representations. Furthermore, we extend this analysis to zero-shot classification following the framework used in previous ablation studies. As shown in Table 6, the full model consistently outperforms ablated versions, particularly in low-data settings, with an average AUROC improvement of 2.1%. These findings highlight the critical role of multi-scale supervision in learning robust and transferable ECG representations.
4.3.3. ABLATION ON ECG ENCODER
Table 7 presents an ablation study evaluating the impact of different ECG encoder backbones on the pretraining per- formance of MELP. Specifically, we compare the Wav2Vec 2.0 (Baevski et al., 2020) architecture with the ResNet-18 backbone used in MERL (Liu et al., 2024a). In addition, we assess the effectiveness of using CMSC (Kiyasseh et al., 2021) for pretraining the ECG encoder prior multimodal training. CMSC is patient-speciffc contrastive learning method for unlabeled ECG data and is orthogonal to the choice of ECG encoder architecture. The results show that employing Wav2Vec 2.0 as the ECG encoder consistently outperforms the CMSC-based variant across all evaluation settings, with an average gain of +3.53%. Furthermore, it outperforms the ResNeet-1-bbsed variant in 17 out of 18 settings, with average improvement of +3.63%.
4.3.2. ANALYSIS ON ECG REPORT GENERATION
Although MELP is primarily designed to encode general- ized global ECG representations, we further evaluate its fine-grained prediction capabilities to validate the effective- ness of our multi-scale pretraining approach. Specifically, we conduct preliminary evaluation the ECG report gen- eration task using the ECGBench dataset (Liu et al., 2024b), and compare our model with PULSE (Liu et al., 2024b), a multimodal baseline trained on ECG images via instruction tuning and evaluated in a zero-shot setting. We employ both standard Natural Language Generation (NIG) metrics
4.3.4. ABLATION ON UNI-MODAL TEXT PRETRAINING
To evaluate the impact of uni-modal text pretraining, we compare the full model with a variant that omits this step. As shown in Table 8, the baseline model (first row) operates without text pretraining, while the full model (second row) incorporates it. The full model outperforms the baseline in
8

---

From Token to Rhythm: A Multi-Scale Approach for ECG-Language Pretraining
Table 10. Ablation results of hyperparameters zero-shot classification tasks.
À1
Aa
PTBXI-Rhythm
PTBXL-Sub PTBXL-For
PTBXL-Super
CPSC2018
CSN
Average
1
1
85.3
66.3
76.9
75.3
82.9
75.6
77.1
1
0.2
84.6
69.0
75.2
76.1
84.0
77.0
77.6
2
1
79.6
68.7
77.1
77.2
83.9
75.2
76.9
2
0.2
85,4
69.1
81.2
76.2
84.2
77.6
78.9
5A
ALS
APB
AF
SVT
TWC
ST
ports learn fine-grained representations for diverse down- stream tasks. Extensive experiments on three benchmark datasets demonstrate its superior performance over existing baselines, highlighting its effectiveness in aligning ECG signals with clinical text at multiple levels of abstraction.
MERL
MELP (Duns)
Limitations and Future Directions. Despite its strengths, MELP has two limitations. First, its token-level supervision lacks explicit clinical interpretability, limiting its applica- bility in explainable diagnostic settings, discussed in Appendix B. Future work will explore leveraging external medical knowledge bases to generate clinically meaningful descriptions, thereby enabling more interpretable and infor- mative token-level supervision. Second, the current model adopts the MedCPT architecture as its text encoder, which does not fully capitalize on the recent advances large language models (LLMs). To address this, future research will investigate ECG instruction tuning frameworks that har- ness the strong generalization and reasoning capabilities of LLMs for enhanced multimodal understanding.
Figure 3. Comparison of T-SNE visualizations of the embedding space for MERL and MELP the CSN test set.
13 out of 18 settings, achieving an average improvement of 0.35%. These results suggest that text pretraining en- hances downstream generalization by improving feature separation in the joint embedding space. They also high- light the potential benefits of developing more specialized cardiology- specific language models to further improve per- formance.
4.3.5. ABLATION ON HYPERPARAMETERS
A ckknowledgment
We evaluate the impact of hyperparameters Åι and A zero-shot classification performance, as shown in Table 10. Overall, the performance remains consistent across four different hyperparameter configurations, demonstrating the robustness of our framework. Through preliminary search, we identify A 2 and A2 0.2 as the optimal setting, which we adopt as the default.
This work was supported in part by the Research Grants Council of Hong Kong (27206123, C5055-24G, and T45-401/22-N), the Hong Kong Innovation and Technol- ogy Fund (TTS/273/22, ITS//74/22, and GHP/318/22GH 1/22GDD, the National Natural Science Foundation of China (No. 62201483), and Guangdong Natural Science Fund (No. 2024N1515011875..
4.4. Qualitative Results
Impact Statement
To analyze the learned representations, we visualize the embedding space of the CSN test set in Figure 3. Follow- ing (Liu et al., 2024a), we focus on seven common ECG abnormalities and select samples that exclusively exhibit each condition. We extract embeddings using both MERI. and MELP and project them into a lower-dimensional space for visualization. The results show that MELP produces more distinct and well-separated embedding space com- pared to MERL. This improved separability helps explain MELP's superior zero-shot classification performance.
The pretraining of MELP is based on large-scale ECG- text pairs. In this study, we utilize the publicly available MIMIC-IV-ECG dataset, which has undergone rigorous de-identification procedures. Nonetheless, privacy consider- ations remain critical, especially when deploying the frame- work in other scenarios. Moreover, although MELP does not explicitly incorporate sensitive attributes such as gender or race during pretraining, its predictions may still reflect subtle biases related to these factors. We encourage future work to further investigate privacy implications and develop effective bias- fias-mitigation strategies to ensure the ethical and equitable deployment of AI systems in healthcare. ad- dition, adapting our framework for clinical scenarios will require careful consideration of regulatory requirements.
5. Conclusion
We introduce MELP, a multimodal ECG foundation model that leverages multi-scale supervision from clinical text re-
9

---

From Token to Rhythm: A Multi-Scale Approach for ECG-Language Pretraining
References
tations. In Machine Learning for Health, pp. 156-167.
PMLR, 2021.
Baevski, A., Zhou, Y., Mohamed, A., and Auli, M. wav2vec 2.0: A framework for self- supervised learning of speech representations. Advances in neural information process- ing systems, (112444924660 2020.
Gow, B., Pollard, T., Nathanson, A., Johnson, A., Moody, B., Fernandes, C., Greenbaum, N., Berkowitz, S., Moukheiber, D., Eslami, P., et al. Mimic-iv-ecg- diagnostic electrocardiogram matched subset. Type: dataset, 2023.
Boecking, B., Usuyama, N., Bannur, S., Castro, D. C.,, Schwaighofer, A., Hyland, S., Wetscherek, M., Naumann, T., Nori, A., Alvarez- Valle, J., et al. Making the most of text semantics to improve biomedical visgan-language processing. In European conference on computer vision, pp. 1-21. Springer, 2022.
Grill, J.-B., Strub, F,, Altché, F,, Tallec, C., Richemond, P., Buchatskaya, E., Doersch, C., Avila Pires, B., Guo, Z., Gheshlaghi Azar, M., et al. Bootstrap your own latent-a new approach self-supervised learning. Advances in neural information processing systems, 33:21271-21284, 2020.
Caron, M., Touvron, H., Misra, I., Jégou, H., Mairal, J., Bojanowski, P., and Joulin, A. Emerging properties in self-supervised vision transformers. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 9650-9660, 2021.
Gwon, H., Seo, J., Park, S., Kim, Y.H.. and Jun, T. J. Medical language model specialized in extracting cardiac knowledge. Scientific Reports, 14(1):29059 2024.
Chen, T., Kornblith, S., Norouzi, M., and Hinton, G. A simple framework for contrastive learning of visual rep- resentations. In International conference on machine learning, pp. 1597-1607. PMLR, 2020.
Han, Y., Liu, X., Zhang, X., and Ding, C. Foundation models in electrocardiogram: A review. arXiv preprint arXiv:2410. 19877, 2024.
Chen, X. and He, K. Exploring simple siamese represen- tation learning. In Proceedings of the IEEF/CVF con- ference on computer vision and pattern recognition, pp. 15750-15758, 2021.
He, K., Fan, H., Wu, Y., Xie, S., and Girshick, R. Mo- mentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 9738-9738, 2020a.
Chen, X., Xie, S., and He, K. An empirical study of training self-supervised vision transformers. In Proceedings of the IEEF/CVF international conference computer vision, pp. 9640-9649, 2021.
He, P., Liu, X., Gao, J., and Chen, W. Deberta: Decoding- enhanced bert with disentangled attention. arXiv preprint 20fi::2000036555 2020b.
Devlin, J. Bert: Pre-training of deep bidirectional trans- formers for language understanding. arXiv preprint 2018
Hu, R., Chen, J., and Zhou, L. Spatiotemporal self- supervised representation learning from multi-lead eog signals. Biomedical Signal Processing and Control, 84: 104772, 2023.
Dong, L., Yang, N., Wang, W., Wei, F., Liu, X., Wang, Y., Gao, J., Zhou, M., and Hon, H.-W. Unified language model pre-training for natural language understanding and generation. Advances in neural information process- ing systems, 32, 2019.
Jin, J., Wang, H., Li, H., Li, Pan, J., and Hong, S. Read- ing your heart: Learning ECG words and sentences via pre-training ECG language model. In The Thirteenth International Conference on Learning Representations, 2025, URL https: wwwp//eeeeeeeeeeeeeeeeeeeee.e.we net fortmm? i 1d=6H21Ko087B.
Ebrahimi, Z., Loni, M., Daneshtalab, M., and Gharehbaghi, A. A review on deep learning methods for eeg arrhythmia classification. Expert Systems with Applications: X, 100033, 2020.
Jin, Q. Kim, W., Chen, Q.. Comeau, D. C., Yeganova, L., Wilbur, J., and Lu, Z. Medept: Contrastive pre-trained transformers with large-scale pubmed search logs for zero-shot biomedical information retrieval. Bioinformat- ics, 39(1) bbbta6651, 2023.
Eldele, E.,, Ragab, M., Chen, Z., Wu, M., Kwoh, C. K., Li, X., and Guan, C. Time-series representation learning via temporal and contextual contrasting. arXiv preprint arXiv:2106. 2ttl::206..41122.. 2021.
Kiyassch, D., Zhu, T., and Clifton, D. A. Clocs: Contrastive learning of cardiac signals across space, time, and pa- tients. In International Conference on Machine Learning, pp. 5606-5615. PMLR, 2021.
Gopal, B., Han, R., Raghupathi, G., Ng. A., Tison, G., and Rajpurkar, 3kg: Contrastive learning of 12-lead elec- trocardiograms using payssiooggicall-inspiii augmen-
10

---

From Token to Rhythm: A Multi-Scale Approach for ECG-Language Pretraining
LB, M. Ccs atrial fibrillation guidelines committee: Cana- dian cardiovascular society atrial fibrillation guidelines 2010: Prevention and treatment of atrial fibrillation fol- lowing cardiac surgery. Can J Cardiol, 27:91-97, 2011.
al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748-8763. PMLR, 2021.
Sangha, V., Khunte, A., Holste, G., Mortazavi, B. J., Wang, Z., Oikonomou, E. K., and Khera, R. Biometric con- trastive learning for data-efficient deep learning from clectrocardiograpiic images. Journal of the American Medical Informatics Association, 3 31(4):855-866 2024.
Li, J., Liu, C., Cheng. S., Arcucci, R., and Hong, S. Frozen language model helps eeg zero-shot learning. In Medical Imaging with Deep Learning, pp. 402 415. PMLR, 2024.
Liu, C., Wan, Z., Quyang, C., Shah, A., Bai, W., and Ar- cucci, R. Zero-shot ecg classification with multimodal learning and test-time clinical knowledge enhancement. 2024a.
Siontis, K. C., Noseworthy, A., Attia, Z. I., and Friedman, P. A. Artificial intelligence-enhaneed electrocardiography in cardiovascular disease management. Nature Reviews Cardiology, 18(7):465-45) 2021.
arXiv preprint
Liu, F., Liu, C.,, Zhao, L., Zhang, X., Wu, X., Xu, X., Liu, Y., Ma, C., Wei, S., He, Z., et al. An open access database for evaluating the algorithms of electrocardiogram rhythm and morphology abnormality detection. Journal of Med- ical Imaging and Health Informatics, 8(7):1368-3733, 2018.
Song. J., Jang, J.-H., Lee, B. T., Hong, D., Kwon, J.-m., and Jo, Y.-Y. Foundation models for eeg: Leveraging hybrid supervid learning for advanced cardiac diagnostics. arXiv preprint arXiv:2407.07 10, 2024.
Tian, Y., Li, Z.,, Jin, Y., Wang. M., Wei, X., Zhao, I., Liu, Y., Liu, J., and Liu, C. Foundation model of ecg diagnosis: Diagnostics and explanations of any form and rhythm on ecg. Cell Reports Medicine, 5(12), 2024.
Liu, R., Bai, Y., Yue, X., and Zhang. P. Teach multimodal Ilms to comprehend electrocardiographic images. arXiv preprint arXiv:2410.1 19008, 2024b.
Mattu, A., Tabas, J. A., and Brady, W. J. Electrocardiog- raphy in emergency, acute, and critical care. American College of Emergency Physicians. 2019.
Wagner, P., Strodthoff, N., Bousseljot, R.-D., Kreiseler, D., Lunze, I., Samek, W., and Schaeffter, T. Ptb-xl, large publicly available electrocardiography dataset. Scientific data, 1-1): 2020.
McKeen, K., Oliva, L.,, Masood, S., Toma, A., Rubin, B., and Wang, B. Ecg-fm: An open electrocardiogram foun- dation model. arXiv preprint o1Xiv:2408.05178, 2024.
Wang, N., Feng, P., Ge, Z., Zhou, Y., Zhou, B., and Wang, Z. Adversarial spatiotemporal contrastive learning for electmcardiogram signals. IEEE Transactions Neural Networks and Learning Systems, 35(10):13845-1338 13859,
Na, Y., Park, M., Tae, Y., and Joo, S. Guiding masked repre- sentation learning to capture spatio-temporal relationship of electrocardiogram. arXiv preprint arXiv:2402.09450, 2024.
2024. doi: 10. (0.1109/TNNISS:22332222111
Williams, R. and Zipser, D. A learning algorithm for con- tinually running fully recurrent neural networks. Neural computation, 1(2):270-280, 1989.
Nie, Y., Nguyen, N. H., Sinthong, P., and Kalagnanam, J. A time series is worth 64 words: Long-term forecasting with transformers. arXiv preprint arXiv:2211.144: 147300 2022.
Wu, Z., Xiong, Y., Yu, S. X., and Lin, D. Unsupervised feature learning via non parammrrett instance discrimina- tion. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 9733-3742, 2018.
Oh, J., Chung, H., Kwon, J.-m., Hong, D.-g., and Choi, E. Lead-agnostic self-supervised learning for local and global representations of electrocardiogram. In Confer- ence on Health, Inference, and Learning, pp. 338-353, PMLR, 2022.
Yan, G., Liang, S., Zhang, Y., and Liu, F. Fusing trans- former model with temporal features for eeg heartbeat classification. In 2019 IEEE International Conference on Bioinformatics and Biomedicine (BIBM), pp. 898-905. IEEE, 2019.
Oord, A. d., Li, Y., and Vinyals, O. Representation leam- ing with contrastive predictive coding, arXiv preprint /807.00307 2018
Yu, H., Yang, H., and Sano, A. Eeg-sl: Flectrocardiogram (eeg) segment learning, deep learning method for ecg signal. arXiv preprint 2x.v::3310.00:: 2023.
Pham, M. Saced, A., and Ma. D. C-melt: Contrastive enhanced masked auto-encoders for ecg- language pre- training, arXiv preprint arXiv:2410.02133, 2024.
Yu, H., Guo, P., and Sano, A. Eeg semantic integrator (esi): A foundation ecg model pretrained with llm-ennanced car- diological text. arXiv preprint arXiv:2405. 19366, 2024.
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J.,

---

From Token to Rhythm: A Multi-Scale Approach for ECG-Language Pretraining
Yu, J., Wang, Z., Vasudevan, V., Yeung, L., Seyedhosseini, M., and Wu, Y. Coca: Contrastive captioners are image- text foundation models. arXiv preprint arXiv:2205.01917, 2022.
Yue, Z., Wang, Y., Duan, J., Yang, T., Huang, C., Tong, Y., and Xu, B. Ts2vec: Towards universal representation of time series. In Proceedings of the AAAI Conference Artificial Intelligence, volume 36, pp. 8980-8987, 2022.
Zbontar, J., Jing, L., Misra, LeCun, Y., and Deny, S. Barlow twins: Self-supervised learning via redundancy reduction. In International conference machine learn- ing. Pp. 12310-1222 12320. PMLR, 2021.
Zhang. W., Yang. L., Geng, S., and Hong. S. Self-supervised time series representation learning via cross reconstruc- tion transforner. IEEE Transactions Neural Networks and Learning Systems, 2023.
Zhao, Y., Zhang, T., Wang. X., Han, P., Chen, T., Huang, L., Jin, Y., and Kang, J. Ecg-chat: A large ecg-language model for cardiac disease diagnosis. arXiv preprint arXiv:2408. 08849, 2024.
Zheng, J. Guo, H., and Chu, H. large scale 12-lead clectrocardiogram database for arrhythmia study (version 1.0. 0). PhysioNet 2022Available online htpphyysione orgcontentecy arrhythmial 10 Oaccessed on, 23, 2022.
12

---

From Token to Rhythm: A Multi-Scale Approach for ECG-Language Pretraining
A. Details of Training Cardiology Language Model
A.1. Cardiology Corpus Details
To extract cardiology-related data from PubMed and Wikipedia, we followed the procedure used in HcartBERT (Gwon al., 2024). For PudMed dataset, we compiled list of cardiology-relater journal names from the SJR (Scimago Journal & Country Rank) database, along with glossaries from Aiken, NIH, and the Texas Heart Institute. These terms used queries retrieve relevant content via the PudMed datahase API. To ensure content relevance, only employ abstract section for PudMed dataset. For Wikipedia dataset, since it already provides information about categories and subcategories for classification, we use a top-level category called "Cardiology" as the primary category. Starting with the "Cardiology" category, we navigated through the subcategories provided by Wikipedia to collect related articles. To ensure content relevance, we only employ abstract section for PudMed dataset. This process finally resulted in curated dataset of approximately 5.6 GB, containing 912.5 million corpus. addition to curated corpus introduced Section A.I, we further incorporate ECG-related reports from the MIMIC-IV-ECG training enhance the model's understanding of ECG-specific clinical language.
A.2. Training Details
Figure illustrates the overall training workflow of our cardiology- speccffcc language model pretraining. We initialize our model using the query encoder from MedCPT (Jin al., 2023)', which was originally trained PubMed search logs. To better adapt it to the cardiology domain, further fine-tune this encoder using masked language modeling (MLM) objective. Specifically, we apply random masking to tokens in ECG diagnostic reports and train the model to reconstruct the masked tokens based on their surrounding context. This encourages the encoder to capture nuanced, Nomainn-specine semantics from cardiology narratives.
Sentences in Reports
Text report datasets
sinus rhythm.
PubMed
Tokenize
split
multiform ventricular premature complexes.
& Random Mask
MIMICIV
Wikipedia
rbbb and lpfb.
Masked Sentence
Masked Sentence
CLS
Tok 1
Tok N
SEP
Tok
Tok M
Egis
ETOk1
TokN
ESEP
T
T
MedCPT query encoder
Egus
E
ETokN
EsEP
Tak1
E
EToKM
Figure 4. Framework for text uni-modal pretraining. Three widely-used medical datasets were combined and segmented into sentences as model inputs. Following BERT-style masking methodology (Devlin, 2018), randomly mask portions of the input text and insert special tokens between sentences. The masked sequences are then processed through the McdCPT query encoder to generate text representations.
13

---

From Token to Rhythm: A Multi-Scale Approach for ECG-Language Pretraining
B. Discussion on Multi-Scale ECG Interpretation
While some portion of ECG reports contain high-level summaries like "sinus rhythm,"" " many real-world diagnostic reports also include detailed references to waveform- features. To further support this, we provide representative examples from our pretraining dataset, MIMIC-IV-ECG database (Gow et al., 2023) in Table 11. These demonstrate that detailed morphological patterns at the waveform (token) level are frequently described in the reports.
Table 11. Examples of ECG reports from MIMIC-IV-ECG dataset. Each row a complete text report and those parts bold are descriptions about fine-grained ECG details.
Regular rhythm. Lead(s) unsuitable for analysis: V1. Q waves in inferior leads. T wave inversion also present. Possible inferior infarction age undetermined. Anterolateral ST-T changes. Summary: abnormal ECG.
Sinus tachycardia. Short PR interval. Borderline ECG.
Sinus rhythm. Poor wave progression - probable normal variant. Anterolateral T wave changes may be due to myocardial ischemia. Abnormal ECG.
Atrial fibrillation. Extensive ST-T changes are nonspecific. Abnormal ECG.
Probable accelerated junctional rhythm. Low QRS voltages in limb leads. Abnormal ECG.
Furthermore, even broader assessments like "sinus rhythm" are based on a set of well-establishen low-level criteria, such as the presence of a P wave before every QRS complex, upright waves in leads I, II, and aVF, regular R-R intervals, consistent PR intervals, and a normal heart rate. More examples can be found in Table 12. Although these features are not explicitly mentioned in the cardiology reports in the MIMIC-IV-ECG dataset, these diagnosis are actually derived from these fine-grained waveform analysis and can be effectively captured by token-level representations during the learning process.
Table 12. Cardiology examples
Clinical diagnosas
oeg crtteria
whether contain deseriptions
complexes leads II, HII aVI, with small R waves and deep
True (Token-level)
Left Anterior Fascicular Block (I.APR)
qR complexes in leads 1, aV,, with small Q waves and tall R waves
Thue (Token-level)
Left Axis Deviation (LAD): Leads and aVF are NEGATIVE, Leads and aVL POSITIVE
False (chythmnlovee)
Irregularly irregular rhythm
False (chythm-leve)
No waves
True (Token-level)
Atrial Fibrillation
QRS complexes usually <120ms
Truc (Token-lovel)
Variable ventricular rale
False (chythm-level)
QRS duration 1200ss
True (laken-level
Dominant S wave in V1
Thue (Roken-level)
Left Bundle Branch Block (LBBB)
Broad monophasic wave lateral leads (1, aVL. V5-6)
(Token-level)
Abseuce of waves in lateral leads
True (Token-level)
Prolenged wave peak time 660mm leads V5-6
(Token-level)
Regular thythm rate of 60- 100 bpm
False (chyth--leve))
Each QRS comples preceded by a normal wave
True (beat-level)
sinux rhythm
Normal wave axis: waves upright in leads and II, inverted in aVR
True (token-level)
The PR interval remains constant
True (token-leval)
QRS complexcs 100
(token-level)
Building on these observations. we argue that using token-level ECG embeddings for report generation not only appropriate but essential for capturing the full spectrum of clinically meaningful information. To clarify this motivation, we have revised the manuscript and included supporting examples that illustrate the rationale behind our design choice.
Moreover, we believe that incorporating more fine-grained textual descriptions-such as explicit references to waveform components-allnssidee general diagnostic terms could further strengthen our approach. These detailed references would provide richer supervision signals and more closely align with established clinical diagnostic criteria. For instance, broad terms like "sinus rhythm"" could be supplemented with explicit criteria such as the presence of P waves before each QRS complex and consistent PR intervals. Integrating such detail into the training data may improve alignment between ECG signals and textual descriptions. While this direction holds promise, it is beyond the scope of the current work and is left for future exploration.
C. Additional Experimental results
C.1. Results for Patient Identification task
The patient identification task trains a model to recognize individual patients based on their ECG signals. It does this by learning to generate unique numerical representations for each person's ECG. The core objective is to ensure ECGs from the
14

---

From Token to Rhythm: A Multi-Scale Approach for ECG-Language Pretraining
Table 13. Zero-shot Patient identification results using Top-k recall [%]. Here we 608 pairs for patient identification following Oh et al. (2022).
Method
PTBXL R@5
R@]
R@ 10
Wav2Vec 2.0 + CMSC RLM (Oh et al., 2022)
39.8
52.14
59.21
ECGFM (McKeen et al., 2024)
49.18
60.70
67.76
MERL (Liu al., 2024a)
16.12
26.32
31.74
MELP
49.67
66.12
70.89
same patient produce highly similar representations, while ECGs from different patients produce distinct ones. For detailed experimental procedures, we follow the settings of (Oh et al., 2022). As shown Table 13, our model Mol.Pouupersorms all three baselines top-1, top-5, and top-10 10 accuracy.
Table Distribution of number beats the training set MIMIC-IV-ECG.
Count
10
15
16
20
Others
Frequency
18357
47635
93075
112010
112424
93830
74027
62150
47997
24509
15987
11606
8347
23493
Percentage
2.5%
6.4%
12.5%
15.0%
15.1%
12.6%
9.9%
8.3%
6.4%
3.3%
2.1%
1.6%
1.1%
3.2%
Table 15.
Ablation results of the number of learnable beats.
PTBXL- PTBXI-Rhythu
PTBXL-Fom
PTBXL-SSu
PTBXL-Super
CPSC2018
CSN
Heats
10%
100%
1%
10%
10%
100%
10%
100%
100%
Average
10
88.87
94.66
96.91
63.41
76.71
83.30
79.22
84.40
N7.46
85.82
87.51
87.87
H8.54
91.75
94.32
78.25
84.83
90.17
85.78
87.98
94.73
96.81
62.33
76.94
84.35
79.69
84.99
86.86
85.61
87.57
87.79
88.58
92.70
93.76
79.89
87.22
86.00
14
87.12
95 82
96.53
64,11
78.92
84.80
80.30 30
85.98
87.31
85. 39
87 47.40
87.66
87.5%
92. 84
94. 14
79.
87.87
91.50
86. 35
97.32
64.74
76.91
83.21
80.66
85.17
87.03
85.56
87.48
87.49
89.18
93.15
94.07
78.91
87.18
90.23
86.41
18
88.01
96.48
97.20
62.71
77.51
83.98
80.37
84.99
86.88
83.68
87.63
87.93
87.65
92.75
93.74
78.48
87.23
91.25
86.
C.2. Ablation on Number Beats.
In the main text, we have chosen heart beats is because we assume that the majority coherts in MIMIC-IV-ECG database have 10 heart beats for each ECG recordings. (The sample duration is 10 seconds and normal heart beat beat per second). However, we have carefully calculate this statistics and found that median number of heart beats 12 13, as evidenced by Table 14. Thus, we have conducted further ablation study to explore more insights of this hyperparameter, and the results are as below in Table 15: The model with 16 learnable heart beats performance best among all variants and have a performance gain of +0.66% compared with 10 learnable beats setting.
D. Dataset Details.
Table 16 summarizes the key statisties of the datasets used in this study. The MIMIC-IV-EGG dataset provides large corpus of 760,618 ECG records without explicit diagnostic labels, serving unlabeled data for pretraining. In contrast, PTB-XL, CPSC2018 8, and CSN are labeled datasets used for downstream evaluation. PTB-XL comprises 12,978 training. 1,642 validation, and 1.652 test samples, spanning five diagnostic categories such as NORM, CD, and HYP. CPSC2018 includes 8,958 training, 1,303 validation, and 2.598 test samples across nine rhythm- and caphhologyrelateed classes, including AF, PAC, and RBBB. CSN consists of 7,651 training, 851 validation, and 2,126 test samples, with annotations for four diagnostic categories such as AF and GSVT. Together, these datasets encompass a broad spectrum of clinically relevant cardiac conditions, enabling robust assessment of model generalization across diverse domains.
E. Implementation Details
E.1. Evaluation Metric
AUC (Area Under the Curve) quantifies the overall classification performance as the area under the ROC curve. Ranging between and 1, a higher AUC indicates superior model discriminative power Specifically, it estimates the probability that the model ranks a randomly selected positive instance higher than a negative instance. Our optimization objective prioritizes AUC maximization achieve optimal TPR-FPR trade-offs.
15

---

From Token to Rhythm: A Multi-Scale Approach for ECG-Language Pretraining
Table 16. Detailed statistics of the datascts.
Split
PTB-XI.
CPSC2001
CSN
NORM
C1)
NORM
L8BH
511
Total
GSVV
Train
7251
2018
1353
116
1907
12978
1213
1289
839
243
1964
864
1081
1148
264
8958
1583
16.99
2804
1625
7651
234
1642
101
33
Test
913
256
184
243
1652
368
342
251
56
589
274
345
2598
449
472
769
436
2126
Table 17. Domain transfer category matching.
Target Category
PTBXI-Super CPSC2018
CSN
AFIB
AFIB
AFIB
APB
PAC
CLBBB
CD
CRBBB
CD
HYP
HYP
RVH, LVH
LBBB
CD
CLBBB
MI
MI
MI
NORM
NORM
NORM
SR
PAC
PAC
SR
NORM
SR
STD
STTC
STD
STE. STTC. STTU, STDD
STE
STTC
STE
STE
STTC
STTC
STTC, STE, TWO, STTU, QTIE, TWC
VPC VPB
VPC
VPB
LAVB
CD
LAVB
IAVB
2AVB / AVB
2AVB, 2AVB1, AVB
RBBB
CD
CRBBB
RBBB
Table 18. Network architecture of MELP.
Block Name
Layer Number
Layer Components
ECG Transformer Encoder
8
MultHHcadAttention. Dropout, LayerNorm, FC layer, LayerNorm
ECG Feature Extractor
4
Convid, Dropout, Fp32GroupNorm, GELU Convid, SamePad, GELU MultiheudAAtennionn. LayerNorm
ECG Positional Encoding
I
ECG Attentional Pooler
1
Text Attentional Pooler
1
MultühendAttention,, LayerNorm
Text Encoder
12
BertAttention, BertUntermeiitt, BertOutput
Text Projector
Linear, GELU, Linear
Text Decoder
6
Residbal/AtemntionBlekk, LayerNorm. MultiheadAttention,, Identity, LayerNorm, MLP, Identity
F.2. Domain Transfer Experimental Details
Table 17 provides the label mappings used for domain transfer experiments. We adopt the SCP-code label alignment protocol as proposed (Liu al., 2024a). Specifically, we conduct transfer learning by training on one dataset and evaluating on another, using the aligned target labels assess generalization under domain shifts, Since MELP supports zero-shot classification, transfer learning is performed by directly evaluating its predictions the overlapping label set between source and target datasets. Categories without direct correspondence are excluded from evaluation to ensure consistency and fairness in performance comparison.
E.3. Network Architecture
Table 18 presents the detailed network architecture of MELP, which comprises modules for both the ECG encoder and the text encoder. The ECG encoder includes an 8-layer Transformer Encoder with multi-head attention, dropout, layer normalization, and feedforward layers, designed to model long-range temporal dependencies. is preceded by an ECG Feature Extractor composed of 4 blocks, each containing Conv Id, dropout, GroupNorm, and GELU activation. Positional information is encoded using a dedicated Conv (-- b-saedd Positional Encoding block. To extract semantically meaningful pooled embeddings, both the ECG and text branches include an Attentional Pooler built with multi-head attention and layer normalization.
The text encoder consists of 12 12-layer Transformer, followed by lightweight Text Projector for dimensionality alignment. Notably, we implement causal attention in the text encoder to prevent information leakage. A 6-layer Text Decoder integrates residual attention blocks and multi-head attention mechanisms to produce cross-modas outputs.
16


Reviews:
[{"review_id": "uas5skzie7", "reviewer": "Reviewer_dLmC", "summary": "The authors propose Multi-scale ECG-Language Pretraining (MELP), which is a two-step process: First is the cardiology language pretraining step, which pretrains a text encoder using cardiology-focused corpus to maximize the language model’s utility for cardiology. Second step is the multimodal pretraining step, which integrates three levels of cross-modal supervision (token, beat, rhythm) for ECG-language pretraining.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the thoughtful and detailed review. **All responses have been updated in our revised manuscript**.\n\n**[A.1] Support for Multi-level Motivation**\n\nThanks for your suggestion. From prior work, we have found several supports for diagnosing ECG by multi-level observations:\n\n-   **Token-level**: Atrial Fibrillation is identified by the absence of P waves and QRS duration <120ms [1].\n-   **Beat-level**: Sinus rhythm requires each QRS complex to be preceded by a normal P wave [2].\n-   **Rhythm-level**: Left Bundle Branch Block diagnosis requires Variable ventricular rate [2].\n\nAs such, we think our proposed multi-scale ECG pretraining is reasonable. We have also incorporated more support in our manuscript.\n\n**[A.2] Additional Compared Baselines (Q1)**\n\nThanks for your comment. As you suggested, we have compared with Wave2Vec 2.0 + CMSC + RLM (Oh et al., 2022) and ECGFM (McKeen et al., 2024) in below Table A.1. As shown, MELP consistently outperforms these two approaches across all experimental settings.\n\n*Table A.1*\n| Method | | PTBXL-Form | | | CPSC2018 | | | CSN | |Average |\n|--------------------------|----------------|------|------|---------------|------|------|--------------|------|------| ------|\n|   | 1%   | 10%   | 100%  | 1%   | 10%   | 100%  | 1%   | 10%   | 100%  | | \n| Wave2Vec 2.0 + CMSC + RLM | 52.72         | 67.81| 80.72| 75.70       | 88.16| 92.61| 65.65  | 78.82| 87.87|    76.67     |\n| ECGFM  | 60.95         | 74.99| 85.54| 82.18       | 89.52| 93.26| 71.51  | 83.17| 88.89|   81.11      |\n| MELP            | **63.41**  |**76.71**|**83.30**| **88.54** |**91.75**|**94.32**| **78.25**|**84.83**|**90.17**| **83.46** |\n\n**[A.3] Justification of ECG Encoder Architecture (Q2)**\n\nThanks for your suggestion. Our understanding is that CMSC is a patient-specific contrastive training approach, and Random Lead Masking is a data augmentation strategy. Neither serves as a network backbone. We have used Random Lead Masking by default in our implementation (see code). We also tested CMSC by pretraining the ECG encoder before multimodal training (as it cannot be directly applied into our framework). As shown in Table A.2, the CMSC variant consistently underperforms, with an average drop of -3.53%. Therefore, we did not adopt it in the final model.\n\n*Table A.2*\n| ECG Encoder | | PTBXL-Form | | | CPSC2018 | | | CSN | |Average | \n|--------------------------|----------------|------|------|---------------|------|------|--------------|------|------|----------------|\n|                       | 1%   | 10%   | 100%  | 1%   | 10%   | 100%  | 1%   | 10%   | 100%   |\n| Wave2Vec 2.0     | **63.41**     | **76.71** | **83.30** | **88.54**     | **91.75** | **94.32** | **78.25**  | **84.83** | **90.17** | **83.81** |\n| Wave2Vec 2.0 + CMSC      | 62.07         | 75.55 | 82.57  | 80.69       | 88.40 | 92.91 | 71.89  | 81.00 | 87.42 |  80.28  |\n\n**[A.4] Additional Downstream Tasks (Q3)**\n\nTo further show the generalizability of MELP, we evaluated it on two tasks: **ECG report generation** and **patient identification**. \n\nWe evaluated report generation using the ECGBench dataset (500 samples), comparing it to the 7B PULSE [3] model. As shown in Table A.3, MELP significantly outperforms PULSE, highlighting its strong fine-grained ECG understanding. Moreover, we evaluated MELP on patient idenfitication in PTB-XL (Table A.4), where it achieved the highest Top-K recall. These results further demonstrates MELP's superior generalizability. \n\nWe agree that ECG question answering is a promising direction, while this task requires text encoder finetuning and we will include in future work. \n\n*Table A.3*\n| Models   | Size | BLEU-1 | BLEU-4 | METEOR | ROUGE-L | BERTScore F1 |\n|----------|------|--------|--------|--------|----------|---------------|\n| PULSE    | 7B   | 5.12   | 0.83   | 13.76  | 8.15     | 10.96         |\n| MELP | 284M | **13.02**  | **1.87**   | **11.28**  | **18.50**    | **44.08**  |\n\n*Table A.4*\n| Method                       | R@1   | R@5   | R@10  |\n|------------------------------|-------|-------|-------|\n| Wave2Vec 2.0 + CMSC + RLM    | 39.8  | 52.14 | 59.21 |\n| ECGFM                        | 49.18 | 60.70 | 67.76 |\n| MERL                         | 16.12 | 26.32 | 31.74 |\n| MELP                     | **49.67** | **66.12** | **70.89** |\n\n**[A.5] Analysis of Number of Beat Tokens (Q4)**\n\nThanks for your valuable question. We initially chose 10 heartbeats based on the assumption of one beat per second in 10-second ECGs. Please see Response [D.3] under Reviewer 5Gq6 for more analysis.\n\n**References**\n\n[1]. Mitchell et al. Canadian Cardiovascular Society atrial fibrillation guidelines 2010: Prevention and treatment of atrial fibrillation following cardiac surgery. Can J Cardiol, 2011.\n\n[2]. Mattu et al. Electrocardiography in emergency, acute, and critical care. American College of Emergency Physicians, 2019.\n\n[3]. Liu et al. Teach Multimodal LLMs to Comprehend Electrocardiographic Images. arXiv preprint 2024.", "claims_and_evidence": "In Section 3.3 Motivation, the authors state “Cardiologists interpret ECG signals in a hierarchical manner, analyzing features at multiple scales – from individual waveform components (tokens) to heartbeats (beats) and overall rhythm”. This claim must be supported with appropriate citation from reputable sources, as it is a crucial statement that inspired the proposed model MELP.", "methods_and_evaluation_criteria": "- The proposed method make sense, but the evaluation criteria lacks depth. Specifically, in Section 3.3 Token-view: Learning to Generate Captions, the authors state “Flexibility in adapting to downstream tasks, such as ECG report generation and ECG-based question-answering” as key advantage of using generative pretraining approach. However, no experiments were conducted to utilize this advantage, and therefore it is unclear why this “advantage” is necessary in this current SSL setting that only conducts ECG classification experiments.   \n- The authors mention in Section 2.1 ECG Presentation Learning “Recent efforts (Oh et al., 2022; McKeen et al., 2024) combine contrastive and generative objectives to develop ECG foundation models”. However, the authors state these methods overlook rich semantic correlations between ECG signals and clinical text reports, because the methods are pre-trained on only ECG datasets. In the Experiments section, these recent efforts are not included in the baseline models used for comparison against MELP. Therefore, the author's statement is speculative and not substantiated with quantitative evaluations. \n- In Section 4.1 Implementation Details the authors mention using Wav2Vec 2.0 architecture for the ECG encoder. However, there are no explanations on why the specific architecture is chosen, or comparison between other architectures such as the  CMSC architecture from CLOCS (Kiyasseh et al., 2021) or Wav2Vec 2.0 + CMSC + Random Lead Masking from (Oh et al., 2022, McKeen et al., 2024).  \n- The authors conduct evaluations on multiple datasets and various evaluation settings (zero-shot, linear probing, cross-domain adaptation), but only on the ECG classification task. This does not truly test the model’s generalizability beyond ECG classification, which is a limitation given the strong generalizability claims of hierarchical representation learning.", "theoretical_claims": "No. The paper does not present theoretical claims requiring proof verification. The contributions are empirical rather than theoretical, focusing on the effectiveness of MELP compared to other baseline models in ECG classification.", "experimental_designs_or_analyses": "Yes I checked the validity of experimental designs and analyses, specifically all classification performance results shown in Tables 2-7.  There are no issues.", "relation_to_broader_scientific_literature": "Research on Self-supervised learning (SSL) methods for ECG-related tasks has gained significant attention in recent years due to its potential to enable various downstream tasks without relying on extensive annotated ECG datasets. The contributions of this paper align well with the broader scientific literature.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "Strenghts\n- The paper proposes a novel multi-scale approach for ECG-Language Pretraining\n- The paper is well-written and easy to understand \n- The Ablation Study is conducted thoroughly to answer some questions regarding the effectiveness of multi-scale approach and cardiology language pretraining\n\nWeaknesses \n- As mentioned in Evaluation Criteria, the experimental section lacks depth \n- Some questions remain unanswered such as why only 10 learnable queries are used in the Heart beat-sentence Alignment section", "other_comments_or_suggestions": "There are minor typos \n- Line 148 pretraing -> pretraining \n- Line 427 Table 4.3 -> Table 8", "questions_for_authors": "- Why were works such as Oh et al., 2022; McKeen et al., 2024 not included in the baseline models for evaluation? \n- Why was the Wav2Vec 2.0 architecture chosen for the ECG encoder? Why were architectures such as CMSC (Kiyasseh et al., 2021) or Wav2Vec 2.0 + CMSC + RLM (Oh et al., 2022) not chosen / compared to the Wav2Vec 2.0 encoder? \n- Were any additional downstream tasks conducted beyond ECG classification to show the strong generalizability of hierarchical representation learning? (e.g., Patient Identification) \n- Why were 10 learnable tokens used in Section 3.3 Beat view: Heart beat-sentence Alignment?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "OuaDAzNyZZ", "reviewer": "Reviewer_DjPw", "summary": "This study proposes MELP (Multi-scale ECG-Language Pretraining), which introduces an innovative multi-scale supervision mechanism in the field of ECG pretraining. By integrating cross-modal alignment at the token, beat, and rhythm levels, MELP effectively enhances the feature learning capability of ECG signals. Compared to existing methods, MELP achieves significant performance improvements in zero-shot ECG classification, linear probing, and transfer learning tasks, demonstrating exceptional generalization ability, especially in low-data scenarios.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the thoughtful review and for acknowledging the novelty of our multi-level supervision and the strength of our experiments. We also appreciate your constructive feedback on the token-level design, which we address below. **All responses are updated in the revised manuscript**.\n\n**[B.1] Clarification of Token-level and Beat-level Supervision (Q1)**\n\nThanks for your insightful feedback. While some ECG reports summarize rhythm-level findings (e.g., “sinus rhythm”), many contain detailed waveform-level observations. Below are examples from MIMIC-IV-ECG, with fine-grained observations in bold:\n\n-   Sinus rhythm. **Poor R wave progression** – probable normal variant. **Anterolateral T wave changes** may be due to myocardial ischemia. Abnormal ECG. \n-   Sinus tachycardia. **Short PR interval**. Borderline ECG.\n-   Atrial fibrillation. **Extensive ST-T changes are nonspecific**. Abnormal ECG.\n-   Probable accelerated junctional rhythm. **Low QRS voltages in limb leads**. Abnormal ECG.\n\nHigh-level diagnoses, like \"sinus rhythm\", still depend on low-level indicators such as P wave consistency and PR interval regularity. Table B.1 illustrates more examples of how clinical interpretations often depend on both global and local ECG features. \n\nSince our token-level pretraining uses a GPT-style objective, rather than masked token prediction, to generate full diagnostic reports from token-level ECG embeddings. By providing full waveform features to the decoder, we allow the model to learn these relationships and generate reports with varying levels of granularity. It may encourage the model to analyze local features and implicitly learn these indicators. Thus, our token-level pretraining design could enhance the model’s ability to learn more generalized, fine-grained ECG representations.\n\nAs for beat-level pretraining, it aims to align beat embeddings with corresponding sentences. While general descriptions in sentences may hinder this process, we think that detailed descriptions encourage the model to understand the ECG at a beat-level.\n\nAs shown in the ablation studies in Tables 5 and 6 of the original manuscript, both token-level and beat-level pretraining improve the learned ECG representations, which further supports our claims.\n\nWe also agree that including more detailed descriptions in the ECG report would further enhance token-level and beat-level pretraining. Please refer to Reply [B.3] for more details.\n\n*Table B.1*\n| Clinical Diagnosis                       | ECG Criteria                                                                                     | Local Feature Mentioned |\n|------------------------------------------|--------------------------------------------------------------------------------------------------|------------------------------------|\n| **Atrial Fibrillation**                  | Irregularly irregular rhythm                                                                     | False            |\n|                                          | No P waves                                                                                       | True                |\n|                                          | QRS complexes usually < 120ms                                                                    | True                 |\n| **Sinus Rhythm**                         | Regular rhythm at a rate of 60–100 bpm                                                           | False              |\n|                                          | Each QRS complex is preceded by a normal P wave                                                  | True                  |\n|                                          | The PR interval remains constant                                                                 | True                |\n\n**[B.2] Details of Positive and Negative Pairs (Q2)**\n\nThank you for the helpful question. We train with a batch size of 64 per device across 4 GPUs. Each ECG and its paired report form a positive pair, while all other ECG-report combinations within the mini-batch (i.e., 255 pairs) are treated as negatives for contrastive learning. \n\n**[B.3] Further Discussion of Limitation (Q3)**\n\nThank you for the suggestion. We further clarified motivation of token-level pretraining in Reply [B.1]. Additionally, we agree that incorporating more explicit fine-grained knowledge, such as detailed waveform descriptions for rhythm diagnosis, could provide stronger supervision and better alignment with clinical criteria. For instance, breaking down terms like \"sinus rhythm\" into their underlying features (e.g., P waves before each QRS, consistent PR intervals) may enhance learning. We have explicitly added this discussion to the revised Limitations section.\n\n**[B.4] Not Discussed References and Typos**\n\nThanks for the careful feedback. We have corrected typos and not discussed references in the revised manuscript.", "claims_and_evidence": "The overall argumentation of the paper is relatively clear, and MELP's performance is systematically validated through standardized benchmarks. However, the implementation of token-level pretraining in the paper may not be entirely convincing. The study employs token-level embeddings to predict the masked portion of the corresponding text, yet the text itself provides an overall description of the ECG signal (e.g., \"sinus rhythm\"). As I understand it, if the goal is to learn fine-grained representations of ECG signals, the corresponding text should also contain fine-grained descriptions. The authors need to further clarify the motivation and experimental design for this aspect.", "methods_and_evaluation_criteria": "Yes", "theoretical_claims": "I have checked the theoretical section in the main text and found no issues.", "experimental_designs_or_analyses": "The experimental setup follows the configuration recommended by MERL and is generally reasonable.", "relation_to_broader_scientific_literature": "This paper can be associated with the field of ECG-Text multimodal learning, offering new perspectives for accurate disease diagnosis in the future.", "essential_references_not_discussed": "The paper does not cite METS (Frozen Language Model Helps ECG Zero-Shot Learning, published in MIDL 2023), which is a pioneering work in the field of ECG-Text multimodal learning. METS first proposed an ECG-Text multimodal zero-shot learning approach, making it a crucial reference for understanding this study.", "other_strengths_and_weaknesses": "The originality of this paper lies primarily in its introduction of the multi-scale concept into multimodal pretraining and its subsequent validation of its effectiveness. However, I believe the main weakness of this paper stems from certain motivational issues in token- and beat-level training. The textual reports used in MELP describe the overall ECG signal rather than providing descriptions at the heartbeat or waveform level. This mismatch in granularity may limit the alignment effectiveness of MELP.", "other_comments_or_suggestions": "- In the Related Work section, ST-MEM is incorrectly spelled as ST-EME。\n- Incorrect citation of HeartLang. In page 3, Beat view: Heart beat-sentence Alignment, and Table 1, HeartLang is cited inconsistently in two different ways. Please use the latest citation: https://openreview.net/forum?id=6Hz1Ko087B.", "questions_for_authors": "- The token-level alignment seems to have a granularity mismatch issue. Could you further clarify the motivation behind this design choice?\n- How are positive and negative samples selected for the rhythm-level contrastive loss?\n- If there is indeed a text granularity misalignment issue, could this limitation be explicitly acknowledged in the paper’s discussion of current constraints?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5FWUZzHkFT", "reviewer": "Reviewer_VGx7", "summary": "This paper proposes a multimodal self-supervised pretraining method for paired electrocardiograms (ECGs) and text. This method, MELP, is unique in its use of multi-scale representation learning and supervision by breaking down an ECG signal into hierarchical levels of the full rhythm view, the smaller beat view, and the smallest token view. Comprehensive experiments show gains over state-of-the-art ECG-language models on a variety of interpretation tasks under varying amounts of labeled data, ranging from zero-shot to fully supervised settings.\n\n## Update after rebuttal\nI will maintain my original recommendation of acceptance.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for the thoughtful evaluation and for recognizing the novelty of our multi-scale ECG-language pretraining approach. We also appreciate your positive remarks on the clarity of our writing and the thoroughness of our ablation studies. Below, we address your concerns regarding the experimental section in detail. **For each point, we have updated our manuscript accordingly.**\n\n**[C.1] Source of Baseline Results (Q1)**\n\nThank you for your helpful question. The baseline results reported in Tables 2 and 3 are cited from MERL [1], which is a standardized benchmark by pretraining its model along with 10 existing self-supervised approaches on the MIMIC-IV-ECG dataset. To be fair, we strictly followed the same experimental setup to ensure a fair comparison. Specifically, we adopted the same pretraining dataset, dataset splits, preprocessing pipeline, and fine-tuning hyperparameters as provided in the official MERL GitHub repository. To verify the reported results, we reproduced the MERL model using their released code and pretrained weights (as seen in Table C.1). Our reproduced results were consistent with those in the original paper, supporting their reliability. In some cases, our reproduced performance was slightly lower, likely due to differences in hardware or software environments. To ensure fairness, we report the original (higher) results from the MERL paper in our comparisons. We will clarify this point in the revised manuscript and include the updated explanation accordingly.\n\n*Table C.1*\n\n|   |  | PTBXL-Rhythm |  | | PTBXL-Form | | | PTBXL-Sub | | | PTBXL-Super | | | CPSC2018 | | | CSN | |Average | \n|------------------------|---------------------|----------------------|-----------------------|------------------|-------------------|--------------------|-----------------|------------------|-------------------|-------------------|-------------------|--------------------|-----------------|-----------------|------------------|-------------|-------------|------------|------------|\n|   | 1% | 10% | 100% | 1% | 10% | 100% | 1% | 10% | 100% | 1% | 10% | 100% | 1% | 10% | 100% | 1% | 10% | 100%  | Average |\n| Reproduced results | 45.33               | 83.92                | 86.13                 | 56.62            | 66.03             | 76.57              | 71.41           | 79.05            | 83.30             | 81.19             | 84.66             | 86.77              | 61.59           | 80.07           | 88.83            | 62.33       | 80.22       | 83.44      |   75.41   |\n| MERL paper         | 52.33               | 82.88                | 88.34                 | 58.26            | 72.43             | 79.65              | 64.90           | 80.56            | 84.72             | 82.39             | 86.27             | 88.67              | 70.33           | 85.32           | 90.57            | 66.60       | 82.74       | 87.95      |   78.05  |\n| Difference         | 7.00                | -1.04                | 2.21                  | 1.64             | 6.40             | 3.08               | -6.51           | 1.51             | 1.42              | 1.20              | 1.61              | 1.90               | 8.74            | 5.25            | 1.74             | 4.27        | 2.52        | 4.51       |   2.64   |\n\n**[C.2] Release of Source Code and Weights (Q2)**\n\nThanks for your great suggestion! To support reproducibility, we will release the full codebase and pretrained model weights to facilitate further research in the community.\n\n**[C.3] Curation of Cardiology-related Corpus (Q3)**\n\nThank you for the comment. We followed the HeartBERT [2] procedure to collect cardiology-related data from PubMed and Wikipedia. For PubMed, we used cardiology journal names and glossaries from SJR, NIH, Aiken, and the Texas Heart Institute to query abstracts via the API. For Wikipedia, we extracted articles under the \"Cardiology\" category and its subcategories, supplemented with glossary-based queries. This resulted in a curated dataset of ~5.6 GB (912.5M corpus). We have added more details in the Appendix of revised manuscript .\n\n**[C.4] Not Discussed References and Typos**\n\nThanks for pointing these out! We have modified typos and updated not discussed references in our revised manuscript according to your suggestions!\n\n**[C.5] Clarification about Methodology Details**\n\nThank you for the positive feedback and helpful suggestions. We have revised the methodology section to improve clarity. \n\n**References**\n\n[1]. Liu et al. Zero-shot ecg classification with multimodal learning and test-time clinical knowledge enhancement. ICML, 2024.\n\n[2]. Gwon et al. Medical language model specialized in extracting cardiac knowledge. Scientific Reports, 2024.", "claims_and_evidence": "Claims appear to be sound, and results are both thorough and convincing at first glance. My only hesitation surrounds the implementation details of baseline methods: were these methods pretrained from scratch on MIMIC-IV-ECG (like MELP, for fair comparison), or were they taken as is (potentially pre-trained on different data) and fine-tuned?", "methods_and_evaluation_criteria": "Yes. This paper leverages large-scale, well-known ECG datasets and evaluation metrics/settings that are consistent with prior literature.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "Experimental design appears sound. Source code has also been provided to aid reproducibility. As mentioned above, my only hesitation concerns whether baseline methods were pretrained from scratch on the same data as MELP or whether they were used “as is”.", "relation_to_broader_scientific_literature": "This study falls as one of a few recent vision-language models released for multimodal ECG-text representation learning. However, it is unique in its multi-scale treatment of ECG representation learning and superior performance compared to relevant state-of-the-art models.", "essential_references_not_discussed": "There are a few areas where more examples of relevant literature could be mentioned, but these are not critical omissions that change the interpretation of results.\n\nE.g., there are additional examples of contrastive methods for ECG [1], reconstruction-based methods for ECG [2], hybrid approaches for ECG [3], and other relevant vision-language foundation models for ECG [4-6]. Jin et al. [6] is the most relevant, particularly because it conceptualizes beats as “words”, but I am aware that this represents a concurrent work.\n\n[1] Sangha, Veer, et al. \"Biometric contrastive learning for data-efficient deep learning from electrocardiographic images.\" Journal of the American Medical Informatics Association 31.4 (2024): 855-865.\n\n[2] Yu, Han, Huiyuan Yang, and Akane Sano. \"ECG-SL: Electrocardiogram (ECG) Segment Learning, a deep learning method for ECG signal.\" arXiv preprint arXiv:2310.00818 (2023).\n\n[3] Song, Junho, et al. \"Foundation Models for ECG: Leveraging Hybrid Self-Supervised Learning for Advanced Cardiac Diagnostics.\" arXiv preprint arXiv:2407.07110 (2024).\n\n[4] Han, Yu, et al. \"Foundation Models in Electrocardiogram: A Review.\" arXiv preprint arXiv:2410.19877 (2024).\n\n[5] Tian, Yuanyuan, et al. \"Foundation model of ECG diagnosis: Diagnostics and explanations of any form and rhythm on ECG.\" Cell Reports Medicine 5.12 (2024).\n\n[6] Jin, Jiarui, et al. \"Reading your heart: Learning ecg words and sentences via pre-training ecg language model.\" arXiv preprint arXiv:2502.10707 (2025).", "other_strengths_and_weaknesses": "*Strengths*:\n- Writing, organization, and presentation are very high-quality and clear to the reader.\n- The multi-scale treatment of ECG signals is unique and appears to be beneficial for downstream performance.\n- Experiments are thorough, with large-scale pretraining and validation on a variety of datasets and tasks to many relevant competitive baselines. Ablation studies help identify which components are most useful.\n\n*Weaknesses*:\n- A few methodological details can be clarified – no obvious weaknesses! Release of source code and model weights will be important to facilitate reproducibility.", "other_comments_or_suggestions": "- Sec 2.1: Title should probably read “ECG Representation Learning” (rather than “Presentation”)?\n- Line 107 RHS: “Yu et al.” is repeated – change this to a parenthetical in-text citation\n- Equation 7: Comma should go inside equation\n- Fig. 1: In the caption, a space is needed before “Token Level”", "questions_for_authors": "1.\tHow specifically were baseline models treated with respect to pretraining? Were they pretrained from scratch on the same data as MELP, or were their weights used “as is” for eventual fine-tuning or zero-shot evaluation? Alternatively, were results in tables ever taken directly from the paper (without the authors running analyses themselves)? Please clarify these details.\n2.\tDo the authors plan to publicly release the code and model weights?\n3.\tIn Sec 3.2, how specifically is “cardiology-related data” extracted or filtered from PubMed and Wikipedia? Include these details in the Supplement.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ZLt3ObYVL2", "reviewer": "Reviewer_5Gq6", "summary": "This paper introduces MELP, a novel multi-modal ECG foundation model that leverages hierarchical supervision at the token, beat, and rhythm levels from clinical text to improve ECG representation learning. \n\nExperimental results on multiple public ECG datasets demonstrate that MELP outperforms existing self-supervised and multi-modal models in tasks such as zero-shot classification, linear probing, and transfer learning.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the thoughtful evaluation and for recognizing the strength of our comprehensive experiments and the novelty of our multi-level supervision design. **We also updated the manuscript accordingly.**\n\n**[D.1] Clarification of Token-level Pretraining (Q1)**\n\nThanks for acknowledging our approach may offer novel insights! From our understanding, token-level embeddings may denote fine-grained ECG features, such as P wave shape, QRS duration, and ST segment changes, by modeling local temporal patterns. Compared with beat-level, it provides more low-level understanding of ECG signals. Although the cardiologists may not explicitly mention token-level features, we think incorporating this level of analysis still encourages deep learning models to better capture the fine-grained characteristics of ECG signals, ultimately improving the model's generalizability.\n\n**[D.2] Ablation on Isolated Variant (Q2)**\n\nThanks for the valuable comment! As the global rhythm level plays a central role in zero-shot ECG classification, we did not initially report isolated module results. To address this, we conducted additional ablation studies evaluating each level independently. Results are shown in Table D.1. These results show that our full model consistently outperforms each isolated variant consistently, confirming the effectiveness of multi-level supervision.\n\n*Table D.1*\n| Loss |  | PTBXL-Form | | | CPSC2018 | | | CSN | |Average | \n|--------------------------|----------------|------|------|---------------|------|------|--------------|------|------|----------------|\n|                       | 1%   | 10%   | 100%  | 1%   | 10%   | 100%  | 1%   | 10%   | 100%  | |\n| $\\mathcal{L}_{\\mathrm{LM}}$       | 52.95 | 63.80 | 76.91  | 64.19 | 73.05 | 85.26 | 69.81 | 79.37 | 84.41 | 72.19 |\n| $\\mathcal{L}_{\\mathrm{Local}}$    |49.81 | 67.82 | 81.41 | 64.18 | 84.08 | 93.17 | 55.89 | 79.77 | *88.79* | 73.88 |\n| $\\mathcal{L}_{\\mathrm{g}}$        |  57.93 | 72.14 | 82.07 | 78.52 | 87.07 | 92.57 | 75.94 | 82.04 | 86.66 | 79.44 |\n| MELP        | **63.41** | **76.71** | **83.30** | **88.54** | **91.75** | **94.32** | **78.25** | **84.83** | **90.17** | **83.48** |\n\n**[D.3] Analysis on Number of Beat Tokens**\n\nThanks for your insightful suggestion! Our initial choice of 10 heartbeats was based on the assumption that most ECG recordings in MIMIC-IV-ECG, with a 10-second duration, would contain roughly one beat per second. However, after analyzing the dataset, we found that the median number of heartbeats per recording is approximately 12–13, as shown in Table D.2.\n\nTo investigate the impact of this hyperparameter, we conducted an ablation study with varying numbers of heartbeat tokens. The results, presented in Table D.3, show that our model is relatively robust to this variation. Notably, using 14 heartbeat embeddings yields slightly better performance than our initial setting.\n\n*Table D.2*\n| Beat Count | 8     | 9     | 10     | 11     | 12     | 13     | 14     | 15     | 16     | 17     | 18     | 19     | 20     | Others |\n|------------|-------|-------|--------|--------|--------|--------|--------|--------|--------|--------|--------|--------|--------|--------|\n| Frequency  | 18357 | 47635 | 93075  | 112010 | 112424 | 93830  | 74027  | 62150  | 47997  | 24509  | 15987  | 11606  | 8347   | 23493  |\n| Percentage | 2.5%  | 6.4%  | 12.5%  | 15.0%  | 15.1%  | 12.6%  | 9.9%   | 8.3%   | 6.4%   | 3.3%   | 2.1%   | 1.6%   | 1.1%   | 3.2%   |\n\n*Table D.3*\n| #.Beats | | PTBXL-Form | | | CPSC2018 | | | CSN | | Average | \n|--------------------------|----------------|------|------|---------------|------|------|--------------|------|------|----------------|\n|                      | 1%   | 10%   | 100%  | 1%   | 10%   | 100%  | 1%   | 10%   | 100%  | |\n| 10      | 63.41           | 76.71 | 83.30 | 88.54      | 91.75 | **94.32** | 78.25 | 84.83 | 90.17 | 83.48 |\n| 12      |  62.33           | 76.94 | 84.35 | 88.58 | 92.70 | 93.76 | **79.89** | 87.22 | 90.29 |  84.01 |\n| 14      |  64.11      | **78.92** | **84.80** | 87.58      | 92.84 | 94.14 | 79.11 | **87.87** | **91.50** | **84.54** |\n| 16      |  **64.74**       | 76.91 | 83.21  | **89.18** | **93.15** | 94.07 | 78.91 | 87.18 | 90.23 | 84.18 |\n\n**[D.4] Further Discussion (Q3)**\n\nThanks for your comment! From our perspective, the contrastive learning excels at learning global representations from ECG. However, its ability to capture fine-grained details is relatively limited. To address this, we have incorporated token-level pretraining based on generation to further enhance the model's detailed understanding for ECG representions. Its benefit can be witnessed in Tables 5 and 6 of our original manuscript, by showing better performance in various downstream datasets. While this added complexity may seem significant, we think it is manageable. Our entire model can be pretrained on 4 GTX 3090 GPUs, with a batch size of 64 per device. We have discussed this point in our revised manuscript.", "claims_and_evidence": "The authors assert that ECG signals have an inherent hierarchical structure with three distinct levels. They argue that clinical text naturally encodes meaningful information corresponding to each of these levels. \n\nThis is motivated by the clinical practice where cardiologists first examine fine-grained waveform details (token level), then group these details into individual heartbeats (beat level), and finally assess the overall rhythm (rhythm level).\n\nAlthough the authors argue that ECG signals possess a hierarchical structure at the token, beat, and rhythm levels, it is difficult to fully embrace token-level learning of ECG and clinical text given that beat-level or lead-level analysis is the standard in ECG interpretation. However, I agree that this approach may reveal novel insights that conventional clinical interpretations might overlook.", "methods_and_evaluation_criteria": "The paper introduces a multi-scale cross-modal pretraining framework that aligns ECG signals with clinical text at three hierarchical levels. \nAt the token level, an encoder-decoder generates report tokens to capture fine-grained waveform details. \nAt the beat level, token embeddings are aggregated into beat representations and aligned with clinical sentences using contrastive learning. \nAt the rhythm level, global ECG representations are created by averaging beat embeddings and aligning them with overall text using a global contrastive loss. \nOverall, the approach is presented in a way that is largely consistent with the intuitive understanding of ECG analysis.", "theoretical_claims": "There isn’t a particularly strong theoretical claim.", "experimental_designs_or_analyses": "The authors present a thorough evaluation of their proposed model on multiple publicly available ECG datasets (e.g., PTB-XL, CSN, and CPSC2018) across various downstream tasks. \n\nThey conduct comprehensive evaluations and extensive ablation studies comparing different configurations. \n\nThe ablation studies involve removing one or more components, with the observed performance drops indirectly confirming the contributions of each level.\n\nHowever, the paper does not offer a fully isolated evaluation of each hierarchical level (token, beat, rhythm). A more granular, independent analysis of each level could further strengthen the empirical support for the model’s design.", "relation_to_broader_scientific_literature": "This paper extends previous multimodal studies that utilize ECG signals and clinical text. \n\nCompared to earlier works, it captures the intuitive perspective of ECG analysis more effectively, and its attempt to evaluate the model fairly using a standardized protocol is particularly noteworthy.", "essential_references_not_discussed": "They referred to recent papers.", "other_strengths_and_weaknesses": "One strength of the paper is its clear and well-written presentation. \n\nHowever, while the authors acknowledge certain limitations, it would have been interesting to see experiments that vary the number of beat quantizations, as this could potentially yield more intriguing results.", "other_comments_or_suggestions": "-", "questions_for_authors": "As mentioned earlier, it would be beneficial if the paper provided further explanation or experimental results on the following two points:\n\n1. Although token-level learning is an innovative approach, it contrasts with the conventional beat-level or lead-level analysis typically used in ECG interpretation. It would be useful to see additional experiments or detailed explanations on how token-level learning compares with standard methods. \n\n2. The current evaluation assesses the integrated contribution of token, beat, and rhythm levels via ablation studies, where one or more components are removed. However, a more granular, independent evaluation of each level would provide clearer insights into how each contributes to the overall performance. Experiments that isolate token-level, beat-level, and rhythm-level supervision could help verify the specific benefits and potential limitations of each component.\n\n3. Many recent ECG representation learning papers include a reconstruction loss alongside contrastive loss to capture both the generative and discriminative features of the data. The absence of a reconstruction loss in the proposed method may be a deliberate design choice. \nWhile reconstruction loss could help retain fine-grained signal details, it might also add complexity and may not be necessary if the contrastive loss sufficiently captures the critical features.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper has received mixed reviews but generally on the positive side. The reviewers generally praised the novelty and empirical strength of MELP across multiple datasets. There did remain some key concerns such as limited evaluation beyond classification tasks (e.g., no demonstration of captioning), lack of clarity around the granularity of token-level alignment with text, and some missing baselines like CMSC and ECGFM, which were addressed in the rebuttal. Overall, I believe the paper is a meaningful contribution to the special area.
Decision: Accept (poster)