Reading Your Heart: Learning ECG Words and Sentences via Pre-training ECG Language Model
Authors: Jiarui Jin
Haoyu Wang
Hongyan Li
Jun Li
Jiahui Pan
Shenda Hong
Venue: ICLR 2025 Poster (2025)
Keywords: Electrocardiogram
ECG
Cardiac signal
Self-supervised learning
ECG language processing
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
TL;DR: We propose HeartLang, a novel ECG self-supervised learning framework that treats heartbeats as words and rhythms as sentences, building the largest ECG vocabulary for ECG language processing to date.

Abstract: Electrocardiogram (ECG) is essential for the clinical diagnosis of arrhythmias and other heart diseases, but deep learning methods based on ECG often face limitations due to the need for high-quality annotations. Although previous ECG self-supervised learning (eSSL) methods have made significant progress in representation learning from unannotated ECG data, they typically treat ECG signals as ordinary time-series data, segmenting the signals using fixed-size and fixed-step time windows, which often ignore the form and rhythm characteristics and latent semantic relationships in ECG signals. In this work, we introduce a novel perspective on ECG signals, treating heartbeats as words and rhythms as sentences. Based on this perspective, we first designed the QRS-Tokenizer, which generates semantically meaningful ECG sentences from the raw ECG signals. Building on these, we then propose HeartLang, a novel self-supervised learning framework for ECG language processing, learning general representations at form and rhythm levels. Additionally, we construct the largest heartbeat-based ECG vocabulary to date, which will further advance the development of ECG language processing. We evaluated HeartLang across six public ECG datasets, where it demonstrated robust competitiveness against other eSSL methods. Our data and code are publicly available at https://github.com/PKUDigitalHealth/HeartLang.

Published as a conference paper at ICLR 2025
READING YOUR HEART: LEARNING ECG WORDS AND SENTENCES VIA PRE-INGGNING ECG LANGUAGE MODEL
Jiarui Jin 1.2.34,4, Haoyu Wang1.4, Hongyan Li Jun Li', Jiahui Pan Shenda Hong*
National Institute of Health Data Science, Peking University
State Key Laboratory of General Artificial Intelligence, Peking University
"School of Intelligence Science and Technology, Peking University
"School of Artificial Intelligence, South China Normal University
hongshenda@pke edu. en, panjiahin @m. scnu, edu.en. leehy@pku.edu.
ABSTRACT
Electrocardiogram (ECG) is essential for the clinical diagnosis of arrhythmias and other heart diseases, but deep learning methods based on ECG often face lim- itations due to the need for high-quality annotations. Although previous ECG self-supereised learning (eSSL) methods have made significant progress in repre- sentation learning from unannotated ECG data, they typically treat ECG signals ordinary sime-series data, segmenting the signals using fixed-size and fixed-step time windows, which often ignore the form and rhythm characteristics and latent semantic relationships in ECG signals. this work, we introduce a novel perspec- tive on ECG signals, treating heartbeats as words and rhythms as sentences. Based on this perspective, we first designed the ORS-Tokenizer which generates seman- tically meaningful ECG sentences from the raw ECG signals. Building on these, we then propose HeartLang, a novel self--upervii learning framework for ECG language processing. learning general representations form and rhythm levels. Additionally, we construct the largest heartbeat- based ECG vocabulary to date, which will further advance the development of ECG language processing. We evaluated HeartLang across six public ECG datasets, where it demonstrated ro- bust competitiveness against other eSSL methods. Our data and code are publicly available at https: http:///ggg..bb..opo/op///
Hea 1 t. HeeaarLLangg
INTRODUCTION
Electrocardiogram (ECG) is common type of clinical data used monitor cardiac ac- tivity, and is frequently employed in diagnos- ing cardiac diseases or conditions impairing myocardial function (Hong et al.) 2020 Liu al. (02T). A primary limitation of us- ing supervised deep learning methods for ECG signal analysis their dependency on large- scale, expert-reviewed, annotated high-quality data. Moreover, even with sufficient data, these methods often designed address specific tasks, which curtails the generalization ability of the model. To overcome these challenges, ECG self-supervised learning (eSSL) has demonstrated efficacy by training on vast amounts of unlabeled ECG recordings to learn generic ECG signal rep- resentations, which are then fine-tuned for specific downstream tasks (Pup & Atzori 2023).
Word
Word
Word
Word
Fixed-size and Fixed-step Time Windows View
Heartbeats Words and
Rhythms as Sentences View (Ours)
Figure 1: Two perspectives on ECG signals.
"quual contribution. Corresponding authors.

---

Published as a conference paper at ICLR 2025
Current eSSL methods can be primarily classified into two categories: contrastive- methods and reconstruction based methods. The core principle of contrastive- methods involves cre- ating positive and negative sample pairs, aiming to maximize the similarity of positive pairs and methods focus
minimize the similarity of negative pairs (Zhang et al. 2023b). Reconstruction-base on training model to reconstruct the original input from partial or transformed data, thus learning effective data representations (Zhang et al., (023c)) However, almost all methods treat ECG signals as ordinary time-series data, which have two significant drawbacks:
Ignoring Form and Rhythm Characteristics of ECG. ECG diagnostics from multi-level char- acteristics are essential Hong et al. (2019). For example, myocardial infarction is diagnosed by observing ST segment elevation of single heartbeat (Vogel et al. 2019). Likewise, cardiac rhythm characteristics are critical, as arrhythmias like atrial fibrillation (AF) are identified based on the overall cardiac rhythm (Carrington et al., 2022). However, existing eSSL methods typically employ fixed-size and fixed-step time windows to segment the signal (Song al.) 2024). This perspective
treats ECG signals ordinary time-series signals, thereby ignoring the unique form and rhythm characteristics inherent to ECG signals, ultimately leading to a decline in the effectiveness of self- supervised learning for both.
Ignoring Latent Semantic Relationships of ECG. Due to significant differences in heart rate and other factors between different subjects, and even among different samples from the same subject (Lan et al. 2022). using fixed-size and fixed-step time windows to segment data leads to substantial discrepancies among samples. The differences between samples disrupt the potential semantic rela- tionships between different heartbeats, which in turn negatively impacts the effectiveness of learning a generalized representation self-supervised learning.
To address these challenges, we propose a self-supervised learning framework named HeartL Lang for ECG language processing (ELP). distinguishing feature of ECG signals is the clear visibility of heart rate patterns, where individual heartbeats are easily identifiable. The core concept of our framework treats heartbeats as words and rhythms as sentences, enabling self-supervised learning at both form and rhythm levels to capture multi-level general representations. Our method consists of four key components: (1) the QRS-Tokenizer, which generates the ECG sentences from the raw ECG signals; (2) the ST-ECGFormes which leverages spatio-temporal information to enhance latent semantic extraction from ECG sentences; (3) the construction of the largest ECG vocabulary to date, where heartbeat quantization and reconstruction enable form-level representation learning; and (4) masked ECG sentence pre-training, which facilitates rhythm-Ievel general representation learning. method can learn both form-level and rhythm-level representations
Through these approaches, of ECG signals without labels, and extract latent semantic representations in ECG sentences. The main contributions of this work summarized in below:
We propose HeartLang, novel self-supervised learning framework for ECG language pro- cessing, designed to learn general representations at form and rhythm levels and extract latent semantic relationships from unlabeled ECG signals.
We present graddigm-shifting perspective of ECG signals, treating them as a language with distinct words (heartbeats) and sentences (rhythms), and design a QRS-Tokenizer that generates the ECG sentences from the raw ECG signals based on this perspective.
We design ST-ECGFormer, novel transformer-based backbone network for ECG signal analysis, which leverages the spatio-temporal features in ECG signals to enhance represen- tation learning and optimize latent semantic relationships extraction for ECG sentences.
To the best of our knowledge, we have constructed the largest ECG vocabulary based on heartbeats date. This ECG vocabulary includes a wide variety of heartbeat morpho- logical representations across different cardiac conditions, which will further advance the development of ECG language processing.
2 RELATED WORK
2.1
SELF-SUPERVISED LEARNING FOR ECG SIGNALS
In recent years, ECG self-supervised learning (eSSL) has demonstrated its ability to learn gen- eral representations from unlabeled ECG signals, significantly improving the performance of down-
2

---

Published as a conference paper at ICLR 2025
stream tasks (Lai et al. 2023). eSSL methods can be broadly categorized into two types: contrastive- based methods and reconstruction-but based methods. For contrastive- based approaches. CLOCS
[Kiyasseh al. 2021 enhances contrastive learning by leveraging cross-space, time, and patient- level relationships in ECG signals, while ASTCI (Wang et al., 2024) employs adversarial learning to capture spatio-temporal invariances in ECG signals. ISI. (Lan al. 2022) enhances cross-subject generalization ability through inter-subject and intra-subject contrastive learning, while BTFS (Yang & Hong 2022) enhances ECG signal classification performance by combining time-domain and methods like MaeFE (Zhang et al., 2023a) and ST-MEM (Na et al.) (024) adopt spatio-temporal approach, learning general ECG representations by masking and reconstructing temporal or spatial content. ECG signals by mutually reconstruct- ing the time-domain and frequency-domain data. However, existing eSSI. methods predominantly focus on apatio-tempore or time-frequency domain representation learning of ECG signals, treating them as ordinary time-series data. This perspective often neglects the morphologically rich semantic information embedded in individual heartbeats.
dremann contrastive learning. On the other hand. reconstruction-bant
CRT /Zang et al., (023c) obtains general representations
2.2
ECG LANGUAGE PROCESSING
ECG language processing (ELP) an emerging paradigm for handling ECG signals, first proposed by Mousavi et al. (02T). Since ECG signals inherently possess significant and clear semantic in- formation in heartbeats, they can be processed using methods similar to natural language processing (NLP). Both Mousavi al. (202I) and Choi et al. 2023) employ approaches that segment different waves within heartbeats to construct vocabularies for modeling. However, when dealing with ECG signals of varying quality, existing methods struggle to accurately segment fine-grained waveforms. Moreover, current ELP methods have relatively small vocabularies (no more than 70 clusters), which limits the richness of the semantic information. In addition, research on ELP remains sparse, high- lighting it as a field in urgent need of further exploration. To address these limitations, we propose a new perspective that directly treats heartbeats as words for modeling and have built the largest ECG vocabulary to date, consisting of 5,394 words, which will significantly advance the development of the ELP research field.
METHOD
In this section, we provide detailed explanation of the specific structure of the HeartLang frame- work. We first define multi-lead ECG data as X E RCXT, where C' represents the number of ECG leads (electrodes) and T represents the total timestamps. The configuration of ECG leads follows the standard 12-lead ECG setup. The overview of the framework is shown in Figure 2 The use of the framework can be divided into four steps. First, a QRS-Tokenizer ECG sentences from the raw ECG signals as described in the Section 3.1 Second, constructing the ECG vocabulary is achieved through the steps in the Section (3.3 Third, masked ECG sentence pre-training of the framework is performed as described in the Section 3.4 Finally, fine-tuning is performed for downstream tasks.
used to generate the
3.1 GENERATING ECG SENTENCES USING THE ORS-TOKENIZER
QRS Detection. A key concept of our method is treat heartbeats words, thus making the segmentation of the original ECG signal into semantic heartbeat patches essential. We introduce Tokenizen, a tokenizer that generates the ECG sentences from the raw ECG signals based on QRS waveforms. Initially, the I-lead signal is bandpass filtered between 5 and 20 Hz, followed by moving wave integration (MWT) using a Ricker wavelet on the filtered signal, and the squared integration signal is saved. The local maxima of the MWI signal are then traversed, with each maximum that occurs after the refractory period and exceeds the QRS detection threshold being classified QRS complex. Following detection, obtain the indices of the detected QRS com- plexes Q = {qili i = I...., N). where N denotes the number of detected QRS indices per sample, which varies between ECG recordings.
Assuming the time window size is t. For each lead, we center each index in Q, using the midpoint between every two indices the interval boundaries, and independently segment the QRS complex
3

---

Published as a conference paper at ICLR 2025
BCG Vecabslary
DOG Sealemae
Individual ECG Words
Lookup
Replace
Collective EDG Wards
Recostrracted
BCG Sentence
QRS Complex Exxexx
Midpoiar
Mulli Lead Segmentt
Spotial Embedding
Temporal Embesking
Randonily ECG Sentence
ST-ECGFamDc Structure
Prodicted Collestive
FOG Wood Indeses
Figure 2: Framework of HeartLang.
patches for each lead. If the segmented region is smaller than l, padding it with zeros match the required size. We refer to these segmented heartbeat patches as individual ECG words, as they are independently extracted from each subject and lack cross-subject generalization.
Generating ECG Sentences. After segmentation, we concatenate the individual ECG words of the 12 leads in sequence, forming the overall ECG sentence gxxt, where represents the sequence length and denotes the time window size. Given the variability in heart rates across samples, the resulting sequence lengths are inconsistent. Similar natural language processing, we set to the maximum length of the ECG sentence. If the length of the ECG sentence is less than I, it will be padded to through zero-filled patches; the length
ECG sentence exceeds I, the interval length will be truncated to l. In this paper, / is set to 256, and is set to 96.
3.2 ST-ECGFORMER BACKBONE NETWORK
To more effectively capture spatio-temporal features and latent semantic relationships within ECG sentences, designed backbone network called ST-ECGFormer. This backbone network is em- ployed in various components of the HeartLang, including vector-quantized heartbeat reconstruction (VQ-HBR) training, masked ECG sentence pre-training, and downstream tasks fine-tuning.
Token Embedding. ECG signals exhibit high temporal resolution, and the QRS complexes that form ECG sentences contain rich temporal features. These QRS complexes are mapped into nigher-dimensional token feature space, allowing their distinguishing features to be more effectively extracted and encoded. We apply a 1-D convolutional layer-based mapping function to transform each individual ECG word into a corresponding token. After this transformation, the ECG sentence can be represented Te E R where denotes the dimension of the token feature space.
Spatio-temporal and Position Embedding, To enable the spatial and temporal information of the ECG sentence to be better captured by HeartLang, a temporal embedding set TE {tea, te,, te2,...................... teio} and spatial embedding set SE {seg, S61, SE2, ... SE12), both D- dimensional and learnable during the training process, were initialized. For the spatial embedding, divide the original 12-lead ECG signals into 12 segments, with each lead corresponding spatial embedding. The spatial embedding of each individual ECG word is mapped back to the lead from which it originated. For the temporal embedding. the original 10-second ECG signal is divided into 10 segments, where each second corresponds to temporal embedding. We assign the temporal embedding of each individual ECG word to the time interval of its QRS complex indices Q. Specif- ically, for zero-filled patches, their spatial and temporal embeddings are set to teg and ség, respec-
4

---

Published as a conference paper at ICLR 2025
tively, to ensure feature consistency. Next, class token is added at the beginning of the sequence to enhance the representation. Additionally, a position embedding list PE = {peu. pê1, peg....,, pe: introduced to reinforee the sequential relationships between individual ECG words. Thus, the ECG sentence can be descrihed by the following formula:
d== Iz + 7 T F + PE,
u {leo, Le,, le2, ... (ee,,..,,e,)): ; u
(seo, se1, 862,.... 8612).
Transformer Encoder. Finally, the ECG sentence will be input into the transformer encoder (Vaswani et al. 2017). To ensure stability during the training process, we employ the pre-layer normalization strategy (Xiong et al. 2020). which applies layer normalization the input of the attention mechanism:
Q = LN K = LN(x''
== LN(2'')',
QK
Attention( Q, K, V) softmax
V,
Vehead
where dhead denotes the dimension of each head in the multi-head attention, and LN represents layer normalization.
3.3
VECTOR- QURRNIZAANEZEDDD HEARTBEAT RECONSTRUCTION TRAINING
The individual ECG words segmented by the QRS-Tokenizer lack generalization properties, as each individual ECG word is independent. We aim for the HeartLang learn general representations across subjects during the subsequent pre-training stage. To achieve this, we introduce ECG vo- cabulary, codebook containing collective ECG words that have cross-subject generalization prop- erties. We believe that the same type of heartbeat across different subjects should be consistent in semantic level. Similar individual ECG words from different subjects are mapped to the same dis- crete and compact collective ECG word, allowing physiological differences between subjects to be overcome and form-level features of heartbeats to be learned. The construction of the ECG vocabu- lary is jointly optimized by quantization and reconstruction processes, depicted in the upper half of Figure 2 This concept is inspired by VQ-NSP (Jiang et al.) 2024), which encodes EEG signals into discrete latent representations and decodes them.
Vector Quantization. We first define an ECG vocabulary V (1,;==,,,..., k: E Rxd where the number of collective ECG words in the vocabulary and d is the dimension of each collective ECG word. Given an ECG signal sample X E RCxT it is first generated by the QRS-To Tokenizer into ECG sentence x /x After the ECG sentence is input into the ST-ECGFormer, a of collective ECG word embeddings / = {p,li ......l .I} is obtained. Then, a quantizer is used to convert them into collective ECG word embeddings. The ECG vocabulary looks up the nearest neighbor of each interval representation in V. We use cosine similarity to find the closest collective ECG word embedding. This procedure can formulated as
2j = arg min [[((())) (2(2x))l2,
where v; the collective ECG word embedding, and Eg represents E, normalization.
Heartbeat Reconstruction. Due to the high signal-to-noise ratio of ECG signals, reconstructing the raw signals directly can efficiently train an ECG vocabulary and effectively learn form-level features of heartheats. After being labeled by the quantizer, the normalized discrete collective ECG are fed into the transformer decoder. This process can be
word embeddings {E2 represented
Ü fa (l2(va))) 1=1
where å the reconstructed ECG sentence and fa the decoder. make the update the ECG vo- cabulary more stable, we adopt an exponential moving average (EMA) strategy. The mean squared error (MSE) loss is utilized to guide the quantization and reconstruction processes. Finally, the loss function for training the VQ-HBR process is defined as
Lv ΣΣ RED i=1
|| - =112 + (2)g((((())) (s +1|e2(p)) sg(62 (o((2(v.))) 11
5

---

Published as a conference paper at ICLR 2025
where DD represents all the ECG sample data, and sg denotes the Kiop-gradiens operation, which is defined as the identity function in the forward pass and has zero gradient.
3.4 MASKED ECG SENTENCE PRE-TRAINING
The pre-tinining process of the HeartLang framework is illustrated in the lower part of Figure 2 In this stage, HeartLang learns generalized rhythm-level representations by masking parts of individual ECG words within an ECG sentence and predicting the corresponding collective ECG words based on the unmasked context. The pre-training stage primarily includes individual ECG words masking and collective ECG words prediction. This process is inspired by the work Peng et al. (2022).
Individual ECG Words Masking, To enable the HeartLang to learn the rhythm-level features of ECG sentences, we perform random masking on the individual ECG words. This allows the HeartLang to understand the content of the entire ECG sentence based on the unmasked context. For the ECG sentence x E R obtained through the QRS- Tokenizer where individual ECG words can be represented as: e = {eeli ........ We randomly generate a mask matrix M = {mm/j
.J. I,...,1). .I.. where m, E (0. 1}. Then, learnable mask token eM E IR is used to replace the masked
heartbeats. Thus, the entire masked ECG sentence can be represented TM {e: m, = 0
......N U {eM m, = 1 = .,
Collective ECG Words Prediction. The task of this stage is to predict the collective ECG word indices of the masked parts based on the unmasked individual ECG words, by minimizing the dis- erepancy between the predicted word indices and the true word indices. We extract the indices of the collective ECG words corresponding to the masked segments of the ECG sentence. The output hidden vectors are then denoted as h (h,
1, ,,...,,, ,1], which are subsequently used predict the corresponding collective ECG words via a linear classifier:
p(v'len) = softmax (Linnar(n))))) (h)).
The target loss function for this stage is:
Lp Cp Σ Σ
logp(>:leM).
ZED m,=1
4 EXPERIMENTS
We follow the benchmark configuration and results provided by MERL (Liu et al. 2024). It worth noting that the baseline methods in the benchmark were tre-trained on MIMIC-IV-CCC (Gow et al.) 2023) dataset. Since MERL is a multimodal method based on ECG-Text, with an additional clinical report text supervision modality, while HeartLang performs pre-training and downstream fine-tuning solely on ECG recordings, we do not directly compare the two methods in the result presentation but instead compare with other eSSL methods.
4.1 VQ-HBR TRAINING AND PRE-TRAINING CONFIGURATION
MIMIC-IV-ECG. This publicly accessible dataset (Gow et al. 2023) contains 800,035 12-lead ECG recordings from 161.352 subjects. Each ECG recording was sampled at 500 Hz and lasted for 10 seconds. To prepare the pretraining dataset, we replaced the "NaN" and "Inf" values in the ECG recordings with the average of six neighboring points.
Implementation. Before VQ-HBR training and pre-training stage, we first downsampled all records in the dataset to 100 Hz and used the Tokenizer transform the raw ECG recordings into a unified ECG sentence. We split the training and validation sets into 9:1, with the validation set data used for VQ-HBR training. In the VQ-HBR training stage, we set the learning rate and trained for 100 epochs, with an ECG vocabulary size of 8,192 and collective ECG word dimension of 128. In the pre-training stage, we set the learning rate to 5 x 10-4, trained for 200 epochs, and applied random masking rate of 50%. For both stages, randomly initialized ST- ECGFormer was used as the backbone network, the AdamW optimizer was selected, and cosine annealing applied for learning rate scheduling, All experiments were conducted on 8 NVIDIA GeForce RTX 4090 GPUs, with a batch size of 64 per GPU. We set the random seed to 0 to ensure the reproducibility of all results. More experimental details are provided in the appendix.
10-5

---

Published as a conference paper at ICLR 2025
Table 1: Linear probing results HeartLang and other eSSL methods. The best results are bolded, with gray indicating the second highest.
PTINNI-Super
PTWXI-Sih
PURNI-FFom
PTRXII-Hhyll
CPSCC20L
CSN
Methed
1%
10%
1001
11%
105
100%
1%
10%
100%
116
10%
1001
1%
10%
100%
116
301%
1001
SinC1. &
Chenet al.
X00yy
61.41
69.77
73.5%
60.80
666 07
71.55 10
54.58
$6.90
52.92
$1.41
09.44
77:75
19.78
65.55
76.54
59.02
57.05
73.30
NYOL
Onller al
N00
71.30
75.03
76.45
$7.15
07 44
71.00
65.75
63.03
10.82
41.99
(74.40
77:17
00:58
74.42
78.75
$4.70
71 92
74.60
Borkew Twins
Zhenur et
2121
72,87
75,06
77541
02.57
70.54
74.34
52.12
00.39
16.14
50.12
73.56
77.02
55.12
72.75
78.59
68.72
71.04
77.43
MoDo V2
Cten et
2021
73.09
76.05
78.26
55.85
09.21
76.05
50.32
63.71
71.31
51.58
71.06
74.33
62.13
76.74
75.29
51.61
74.26
77.68
SinSSam
Ches He
2021
73.15
72.70
75.03
62.52
09.31
76.38
55.16
62.91
71.31
49.30
09.47
75.92
54.35
72.89
75.31
58.25
58.61
77.41
TS-TUC
Eddcle ml.
2001
70.33
75.88
73.91
53.54
66.98
77.87
43.04
61.79
71.15
43.54
09.48
77:22
57.07
73.62
78.72
55.25
58.45
76.79
CLOCS
Kayasccher L1. 200
68.94
73.36
76.31
57.94
72.55
76.34
51.97
57.96
72.65
47.19
71.88
76.31
59.59
77.78
77.49
54.38
71.93
76.13
ASTCL
Wang
1024
72.51
77.31
81.02
61.96
68.77
76.51
44.14
60.93
56.99
$2.58
71.98
76.05
57.90
17.00
79.51
$6.40
70.87
75.79
CRT Zhag
A023a
69.68
18.24
77.24
61.95
70.32
77667
46.41
59.40
68.73
47.44
73.52
74.41
58.01
76.43
62.03
56.21
73.70
78.50
ST-MEM Na.tall clal.
1034
61.12
66.87
71.36
54.12
57.56
63.99
(5.77
59.99
56.07
51,12
65.44
74.85
56.69
63.33
70.39
59.77
56.87
71.36
HeariLang (Ours)
78.94
85.59
87.52
64.68
79.34
85.91
58.70
63.99
50.23
62.08
76.22
90.34
60.44
66.36
77.87
57.94
68.93
82.49
4.2 DOWNSTREAM TASKS CONFIGURATION
We evaluated our method on the three widely used public datasets listed below, which cover over 100 types of cardiac conditions. Detailed information on the data split can be found in the appendix.
PTB-XL. This publicly accessible dataset (Wagner et al., 2020) contains 21,837 12-lead ECG recordings collected from 18,885 patients. Each ECG recording was sampled at 500 Hz and lasted for 10 seconds. Based on the SCP-ECG protocol, the multi-class classification task has four subsets: Superclass (5 classes), Subclass (23 classes), Form (19 classes), and Rhythm (12 classes). We followed the official data split (Strodthoff et al. 2021) for training, validation, and testing.
This publicly accessible dataset (Liu et al. 2018) contains 6,877 12-lead ECG record- ings. Each recording was sampled at 500 Hz, with durations ranging from 6 to 60 seconds. The dataset is annotated with 9 different labels. We split the dataset into 70%: 10%:00%00% for training, validation, and testing.
CPSC2018.
(CSN). This publicly accessible dataset (Zheng et al. 2020 2022) contains 45,152 12-lead ECG recordings. Each ECG recording was sampled at 500 Hz and lasted for 10 seconds. Following the configuration provided by MERL, we removed ECG records with "unknown" annotations. The refined version of the dataset contains 23,026 ECG recordings with 38 distinct labels. We split the dataset into 70%, 10%, 20% for training, validation, and testing.
Soaoman-ShaxxingNinnge
Implementation. Before fine-tuning in downstream tasks, we first downsampled all records in the dataset 100 Hz and used the QRS-Tokenizer to transform the raw ECG recordings into unified ECG sentence. For linear probing, we kept the ST-ECGFormer backhone network frozen and only trained the randomly initialized parameters of the linear classifier. To explore the performance of our method under low-resource conditions, we conducted linear probing using 1%. 10%, and 100% of the training data for each task. We set the learning rate to 5 10-3 and trained for 100 epochs. For the CPSC2018 and CSN datasets, we scaled the ECG recordings to the range of [-3,3] to enhance QRS detection. All test results were obtained from the best validation model, rather than testing the model on the test set after each epoch and reporting the highest result. For all downstream tasks, we used the maero AUC as the evaluation metric. We set the random seed to 0 to ensure the reproducibility of all results. More experimenaal details are provided in the appendix.
RESULTS AND DISCUSSIONS
5.1 EVALUATION ON LINEAR PROBING
Table presents the linear probing results of HeartLang compared to existing eSSL methods. In the PTB-XL dataset, Heartl consistently demonstrated significant advantages across 1 1% 100% of the training data. Specifically, compared to the second-best eSSL method, our method achieved an average macro AUC improvement of 8.14. Notably, on the Form and Rhythm subsets in PTB-XL dataset, HeartLang outperformed the second-best eSSL methods by an average of 9.85 in macro
7

---

Published as a conference paper at ICLR 2025
Hoi Uteals Worls and #tydlers sertinces Theddebee and Toondetpp Time Witcoss
0.330
Jocume
8.330
106
125
tpcch
E3003
Figure 3: The validation loss curve during VQ-HBR training (left) and the prediction accuracy curve during masked ECG sentence pre-training (right), shown from two perspectives.
Table 2: Linear probing results of two signal slicing perspectives. The best results are bolded, while improved values are marked in green, and decreased values are marked red.
PTBXL-Super
PTBXL-Sub
PTBXL-Form
PTBXL-Rhythm
Perspective
1%
10%
100%
1 %
10%
100%
1%
10%
100%
1% 10%
100%
Fixed-size Fixed-step
66.41
77.89 80.51
60.87
70.72
78.32
59.94
65.52
76.03
55.83
74.09
86.05
Time Windows
Heartheats Words and
78.94
85.59
87.52
64.68
79.34
88.91
58.70
63.99
80.23
62.08
76.22
90.34
Rhythms as Sentenees
Improvement
12.52
7.71
7.01
3,81
8.62
10.58
-1.24
-1.53
4.20
6.26
2.14
4,29
AUC with 100% of the training data, further highlighting significant advantages of Heartlang in ECG heartbeat and rhythm representation learning. This validates the effectiveness of our proposed signal slicing perspective of heartbeats as words and rhythms as sentences. For the CPSC2018 and CSN datasets, our method only outperformed others on the CSN dataset with 100% training data. We speculate that this is due to the significant baseline drift in these datasets, which may have reduced the performance of the QRS-Tokenizer. Nevertheless, our method remains highly competitive compared to other eSSL methods.
We attribute the weaker performance of other eSSL methods to their disruption of the semantic information in ECG signals. For contrastive eSSL methods, data augmentation methods such ro- tation, shifting, and adding noise introduce semantic distortion to the positive and negative pairs in ECG signals, which leads to decline in representation learning performance. For generative eSSL methods, treating ECG signals as ordinary sime-series data and applying the fixed-size and fixed- step time windows for slicing cannot accommodate the broad and complex dynamic characteristics of ECG signals. This results in patches without clear semantic information, ultimately leading to method the QRS-Toker
decline in representation learning performance. In contrast, to segment ECG patches with clear semantic meaning and enhances representation learning by re- constructing both heartbeat form and cardiac rhythms, ultimately achieving superior performance in downstream tasks.
5.2
EVALUATION ON SIGNAL SLICING PERSPECTIVE
To further validate the effectiveness of our proposed "heartbeats as words, rhythms as sentences" signal slicing perspective, we compared it with the traditional signal slicing perspective in this ex- periment, which utilizes the fixed window size and time step. For fixed-size and fixed-step time windows perspective, created ECG sentences from the raw ECG recordings in the same manner but without the QRS detection process, instead slicing based on fixed window sizes and strides. We then performed VQ-HBR training, pre-training, and linear probing using the same configurations.
Figure presents the VQ-HBR training loss curves and masked ECG sentence pre-training pre- diction accuracy curves for both perspectives. During the VQ-HBR training stage, our proposed perspective significantly outperforms the traditional signal slicing perspective. This is because the
8

---

Published as a conference paper at ICLR 2025
Different Heartbeat Morphologles Can Couvey Different Semantic Expressions
ST-segment Depression
Normal Heartbeat
Wide QRS Complex
The Largest
ECG Vocabulury
V193
V1795
V5171
V8192
2012
Byck Diace
2011
Similar Heartbeat Morphologles Also Convey Different Semantic Expressions Depending Context
Figure 4: ECG vocabulary visualization.
patches generated by method carry clear semantic information, which benefits the training of the ECG vocabulary. However, in the pre-training stage, the traditional slicing perspective shows higher masked prediction accuracy. We attribute this difference primarily the number zero- filled patches. The ECG sentences generated by the traditional slicing perspective use fixed 120 patches, resulting in a larger number of zero-filled patches. In contrast, the number of patches used in our proposed perspective is dynamic and dependent on the heart rate. The larger number of zero-filled patches in the traditional perspective may improve the performance of masked prediction presents the results of downstream fine-tuning on the PTB-XL dataset
during pre-training. Table 2 using two different perspectives. In the majority cases, our perspective resulted in significant im- provements in macro AUC performance, except for a few low-resource scenarios where there was slight performance drop. Specifically, our perspective resulted in an average improvement 5.36 in macro AUC. Notably, the Superclass and Subelass subset, perspective brought average improvement of 8.38 macro AUC.
Despite the fact that masked ECG sentence pre-training prediction accuracy of our perspective was lower than that of the traditional perspective during the pre-training stage, it achieved better results in downstream tasks. We attribute this to the higher modeling difficulty in masked ECG sentence pre-training for our perspective, which helped the model uncover latent semantic relationships be- tween heartbeats, leading the learning of more generalized representations. Interestingly, in the Superclass and Subclass subsets, using the traditional perspective caused our method to degrade baseline-like performance. We speculate that if baseline methods adopted our perspective, they would also see performance improvements. Furthermore, in the Form and Rhythm subsets, even with the traditional perspective, our method outperformed baseline methods, further demonstrating the effectiveness of the HeartLang architecture in multi-level representation learning of ECG signals.
5.3 ECG VOCABULARY VISUALIZATION
In this section, we visualize the ECG vocabulary to reflect how the collective ECG words corre- spond the original individual ECG words. We trained and validated VQ-HBR MIMIC-IV- ECG, where the effective use of collective ECG words in the validation set amounted to 5,394 (dis- crete words). In Figure 4 the individual ECG words with the same index exhibit similar and clear semantic information in terms of heartbeat representation. For instance, index 1193 displays the morphological features of ST-segment depression, while index $171 shows the characteristics of wide QRS complex. We further visualized ECG sentence segments centered around individual ECG

---

Published as a conference paper at ICLR 2025
Table 3: Linear probing results of the ablation study.
The best results are bolded, with gray
indicating the second highest.
PTBXL-Super
PIBXL-Sub
PTBXL-Form
PTBXL-Rbyttm
Method
1%
10%
100%
1%
10%
100%
1%
10%
100%
1%
10%
100%
w/o Vocabolary
59.23
61.51
79.68
49.34
50.55
46.88
63.25
42.87
43.87
77.65
w/o Pre training
70.34
77.82
80.31
55.70
67.64
80.57
53.73
57.67
66.70
65.98
76.49
81.40
w/o Spatio-tempore temporal Embhediing
69.78
81.79
85.12
58.06
73.76
87.33
55.09
63.37
73.90
61.79
74.83
84,93
w/o Spatial Embedding
78.74
85,32
86.87
67.82
79.31
88,66 66
60.76
67.66
79.10
63.54
79.00
89.25
w/o Temporal Embedding
77.87
84.86
85.85
64.56
77.48
87.50
61.41
67.54
77.50
65
84.74
87.81
HeartLang (Ours)
78.94
85,59
87.52
64.68
79.34
88.91
58.70
63,99
80.23
62.08
76.22
90.34
words three distinct indices, each surrounded by six neighboring individual ECG words (three on each side). In our constructed ECG sentences, the incorporation of spatio-temporal and position embeddings leads similar heartbeats (e.g., normal heartbeat) within the same ECG sentence being mapped to different collective ECG words. This phenomenon suggests that collective ECG words encapsulate spatio-temporal and positional features, providing richer semantic information. Simi- larly, in English, the same word can serve different grammatical functions (e.g., 'run' can be a verb or noun). The same words (similar heartbeat morphology) with identical spellings can assume dif- ferent parts of speech (different collective ECG words) depending on the context (spatio-tempora and position embeddings). However, in previous ELPP studies, vocabularies were only constructed by clustering morphologically similar ECG waves for modeling (not heartbeats, clusters), without incorporating apatio-tempore and positional information, which resulted in lim- ited parts of speech. Expressions of different parts of speech through the context further highlights the richer semantic representation of our study compared to previous ELP studies.
more than 70
5.4 ABLATION STUDY
In this section, we present comprehensive ablation study of HeartLang illustrate the effectiveness of each component within the framework, as shown in Table For the experiment without the ECG vocabulary, we discarded it and applied mean squared error (MSE) loss to reconstruct the masked segments. This approach did not allow HeartLang to effectively learn general representations during pre-training, highlighting the role of ECG vocabulary helping HeartLang capture latent semantic relationships in ECG sentences. For the experiment without pre-training, we fine-tuned randomly initialized ST-ECGFormer on downstream tasks. The results indicated that the pretraining process effectively learned general representations in ECG sentences, leading to significant improvements in downstream performance. We also conducted an ablation study on the structure of ST-ECGFormer, completing all training and fine-tuning procedures. The results showed that, aside from slight performance drop under low-resource conditions, HeartLang consistently enhanced performance, demonstrating that ST-ECGFormer effectively learns spatio-temporal representations of ECG sig- nals and boosts the performance of HeartLang on downstream tasks.
6 CONCLUSION
In this paper, we propose a novel perspective on ECG signal processing, treating them as a language with distinct words (heartbeats) and sentences (rhythms). Based on this perspective, we introduce the QRS-Tokenize
which generates the ECG sentences from the raw ECG signals. Building upon these, we propose HeartLang, a novel self-supervised learning framework for ECG language pro- cessing. HeartLang learns form-level general representations through vector- quantized heartbeat re- construction training and rhythm-level general representations through masked ECG sentence pre- training. Additionally, we constructed the largest heartbeat-based ECG vocabulary to date. This ECG vocabulary includes a wide variety of heartbeat morphological representations across different cardiac condition, which will further advance the development of ECG language processing. We evaluated HeartLang across six public ECG datasets, where it demonstrated robust competitiveness against other eSSL methods. We hope that the ideas presented here can inspire the ECG research community, particularly in the emerging field of ECG language processing.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
This work was supported by the National Natural Science Foundation of China (62102008, 62172018); CCF-Zhipu Large Model Innovation Fund (CCF-Zhipu202414)) STI 2030-Major Projects (202ZZD2008900); and the Guangdong Basic and Applied Basic Research Foundation (2024A1510010524..
REFERENCES
Mafalda Carrington, Rui Providência, Anwar A Chahal, Fabrizio Ricci, Andrew E Epstein, Sabina Gallina, Artur Fedorowski, Richard Sutton, and Mohammed Y Khanji. Monitoring and diagnosis of intermittent arrhythmias: Evidence-based guidance and role of novel monitoring strategies. European Heart Journal Open, 2(6):0eac072, November 2022. ISSN 2752-4191. doi: 10. 10.1093/
chjopen/oeac072.
Ting Chen, Simon Korblith, Mohammad Norouzi, and Geoffrey Hinton. A Simple Framework for Contrastive Learning of Visual Representations. In Proceedings of the 37th International Conference on Machine Learning, pp. 1597-1607. PMLR, November 2020.
Xinlei Chen and Kaiming He. Exploring Simple Siamese Representation Learning. In 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 15745-15753, Nashville, TN, USA, June 2021. IEEE ISBN 978-1-66544-509-2. doi: 10. N0.100//CPR444437. 2021.01549.
Xinlei Chen, Saining Xie, and Kaiming He. An Empirical Study of Training Sulf-Super Vision Transformers. In 2021 IEEE/CVF International Conference Computer Vision (ICCV), pp- 9620-9629, Montreal, QC, Canada, October 2021. IEEE. ISBN 978-1-66552-811255 doi: 10. 1109/CCCV48922.002 202..9999
Scokmin Choi, Sajad Mousavi, Phillip Si, Haben G. Yhdego, Fatemeh Khadem, and Fatemeh Afghah. ECGBERT: Understanding Hidden Language of ECGs with Self-Supervised Repre- sentation Learning. (arXiv:2306.06340, June 2023.
Emadeldeen Eldele, Mohamed Ragab, Zhenghua Chen, Min Wu, Chee Keong Kwoh, Xiaoli Li, and Cuntai Guan. Time-Series Representation Learning via Temporal and Contextual Contrasting, In Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, pp. 2352- 2359, Montreal, Canada, August 2021. International Joint Conferences Artificial Intelligence Organization. ISBN 978-0-9992411 199-6. doi: 00.24963/iccai.2221/222
Brian Gow, Tom Pollard, Larry A Nathanson, Alistair Johnson, Benjamin Moody, Chrystinne Fer- nandes, Nathaniel Greenbaum, Seth Berkowitz, Dana Moukheiber, Parastou Eslami, et al. Mimic- iv-ecg- diigg-dia electrocardiogram matched subset. Type: dataser, 2023.
Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, Bilal Piot, koray kavukcuoglu, Remi Munos, and Michal Valko. Bootstrap Your Own Latent A New Approach to Self- Sulersuuer Learning. In Advances in Neural Information Processing Systems, volume 33, pp. 21271-21288 2224. Curran Associates, Inc., 2020.
Shenda Hong, Cao Xiao, Tengfei Ma, Hongyan Li, and Jimeng Sun. Mina: Multilevel knowledge- guided attention for modeling electrocardiography signals. In 28th International Joint Conference on Artificial Intelligence, IJCAI 2019. pp. 5888-5894. International Joint Conferences on Artifi- cial Intelligence, 2019.
Shenda Hong, Yuxi Zhou, Junyuan Shang, Cao Xiao, and Jimeng Sun. Opportunities and challenges of deep learning methods for electrocardiogram data: A systematic review. Computers in biology and medicine, 122: 103801, 2020.
Weibang Jiang, Liming Zhao, and Bao liang Lu. Large brain model for learning generic represen- tations with tremendous EEG data in BCI. In The Twelfih International Conference on Learning Representations, 2024. URL https: http://pwrrreeeeteeefooroorrr///

---

Published as a conference paper at ICLR 2025
Dani Kiyasseh, Tingting Zhu, and David A. Clifton. CLOCS: Contrastive Learning of Cardiac
Signals Across Space, Time, and Patients. In Proceedings of the 38th International Conference on Machine Learning, pp. 5606-5615, PMLR, July 2021.
Jiewei Lai, Huixin Tan, Jinliang Wang. Lei Ji, Jun Guo, Baoshi Han, Yajun Shi, Qianjin Feng, and Wei Yang. Practical intelligent diagnostic algorithm for wearable 12-lead ECG via self-supervised learning on large-scale dataset. Nature Communications, 14(1):3741, June 2023. ISSN 2041-
1723. doi: 10. 4a0e4442-677---99
Xiang Lan, Dianwen Ng. Shenda Hong, and Mengling Feng. Intra-inter subject self-supervieed learning for multivariate cardiac signals. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pp. 3532-4540 2022.
Che Liu, Zhongwei Wan, Cheng Ouyang, Anand Shah, Wenjia Bai, and Rossella Arcucci. Zero- shot ECG classification with multimodal learning and test-time clinical knowledge enhance- ment. In Forty-first International Conference on Machine Learning, 2024. URL https:
//openzeview. www.eee//toooe -ow.:///ddd22222000-------- 2vJ2210kKj
Feifei Liu, Chengyu Liu, Lina Zhao, Xiangyu Zhang, Xiaoling Wu, Xiaoyan Xu, Yulin Liu, Caiyun Ma, Shoushui Wei, Zhiqiang He, et al. An open access database for evaluating the algorithms of electrocardiogram rhythm and morphology abnormality detection. Journal of Medical Imaging and Health Informatics, 8(7):1368 1373, 2018.
Xinwen Liu, Huan Wang, Zongjin Li, and Lang Qin. Deep learning in ECG diagnosis: A review. Knowledge-Bass Based Systems, 227:107187, September 2021. ISSN 09507051. doi: 10. k00.6....noss.22211107777.
Sajad Mousavi, Fatemeh Afghah, Fatemeh Khadem, and U. Rajendra Acharya. ECG Language processing (ELP): A new technique to analyze ECG signals. Computer Methwds and Programs in Biomedicine, 202: 105959, April 2021. ISSN 01692607. doi: 10. h0t.6/..cmpb.222...10.0 105959.
Yeongyeon Na, Minje Park, Yunwon Tae, and Sunghoo Joo. Guiding masked representation learn- ing to capture spatio-temporal relationship of electrocardiogram. In The Twelfth International Conference on Learning Representations, 2024.
net forr/mrrmm
id=WcOohbsF4F
Jungwoo Oh, Hyunseung Chung. Joon-myoung Kwon, Dong-gyun gyun Hong, and Edward Choi. Lead- agnostic self-supervised learning for local and global representations of electrocardiogram, In Conference on Health, Inference, and Learning, pp. 338-353. PMLR, 2022.
Zhiliang Peng, Li Dong, Hangbo Bao, Qixiang Yc, and Furu Wei. BEiT v2: Masked Image Model- ing with Vector-Quantian Quantized Visual Tokenizers. MurXiv:22080063660 October 2022.
Federico Del Pup and Manfredo Atzori. Applications of Self-Supervised Learning to Biomedical Signals: A Survey. IEEE Access, 11:144180-144203, 2023. ISSN 2169-3536. doi: 10.1 1099
ACCEESS.002.3344533.
Junho Song, Jong-Hwan Jang, Byeong Tak Lee, DongGyun Hong. Joon-myoung Kwon, and Yong- Yeon Jo. Foundation Models for Flectrocardiograms. (arXiv:2407.077 10), June 2024.
D
Nils Strodthoff, Patrick Wagner, Tobias Schaeffter, and Wojeiech Samek. Deep Learning for ECG Analysis: Benchmarks and Insights from PTB-XL. IEEE Journal of Biomedical and Health Informatics, 1519 May 2021. ISSN 2168-2194, 2168-2208. doi: 10.1109/JBHI. 2020.3022989.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is All you Need. Advances in neural informa- fion processing systems (NeurIPS), 30, 2017.
Birgit Vogel, Bimmer E. Claessen, Suzanne V. Arnold, Danny Chan, David J. Cohen, Evangelos Giannitsis, C. Michael Gibson, Shinya Goto, Hugo A. Katus, Mathieu Kerneis, Takeshi Kimura, Vijay Kunadian, Duane S. Pinto. Hiroki Shiomi, John A. Spertus, P. Gabriel Steg, and Roxana Mehran. ST-segment elevation myocardial infarction. Nature Reviews Disease Primers, 5(1):39, June 2019. ISSN 2056-676X. doi: 10. (1188/41572-01-0000
12

---

Published as a conference paper at ICLR 2025
Patrick Wagner, Nils Strodthoff, Ralf-Dieter Bousseljot, Dieter Kreiseler, Fatima I. Lunze, Wojeiech Samek, and Tobias Schaeffter. PTB-XL, a large publicly available electrocardiograapyy dataset. Scientific Data, 7(1):154, May 2020. ISSN 2052-4463. doi: 10. 10.1038//41597-020-0000
Ning Wang, Panpan Feng, Zhaoyang Ge. Yanjie Zhou, Bing Zhou, and Zongmin Wang. Adver- sarial Spatiotemporal Contrastive Learning for Electrocardiogram Signals. IEEE Transactions on Neural Networks and Learning Systems, pp. 1-15, 2024. ISSN 2162-237X, 2162-2388. doi: 10. 10.1109/TNNLSS.2233377777
Ruibin Xiong, Yunchang Yang, Di He, Kai Zheng, Shuxin Zheng, Chen Xing, Huishuai Zhang, Yanyan Lan, Liwei Wang, and Tieyan Liu. On Layer Normalization in the Transformer Archi- tecture. In Proceedings of the 37th International Conference Machine Learning, pp. 10524- 10533. PMLR, November 2020.
Ling Yang and Shenda Hong. Unsupervised time-series representation learning with iterative bilin- temporal-spectral fusion. In International conference on machine learning, pp. 25038-25054 PMLR. 2022.
Jure Zbontar, Li Jing, Ishan Misra, Yann LeCun, and Stephane Deny. Barlow Twins: Self-Supervised Learning via Redundancy Reduction. In Proceedings of the 38th International Conference on Machine Learning, pp. 12310-12320. PMLR, July 2021.
Huaicheng Zhang, Wenhan Liu, Jiguang Shi, Sheng Chang, Hao Wang, Jin He, and Qijun Huang. MaeFE: Masked Autoencoders Family of Electrocardiogram for Self-Supervised Pretraining and Transfer Learning. IEEE Transactions Instrumentation and Measurement, 72:1-15, 2023a. ISSN 0018-9456, 1557-9662. doi: 10.1 1..10//M...002222222...
Wenrui Zhang, Shijia Geng, and Shenda Hong. A simple self- self-supe ecg representation learning method via manipulated temporal -ppatall reverse detection. Biomedical Signal Processing and Control, 19:104194, 2023b.
Wenrui Zhang, Ling Yang. Shijia Geng, and Shenda Hong. Self-Supervised Time Series Represen- tation Learning via Cross Reconstruction Transformer. IEEE Transactions on Neural Networks and Learning Systems, pp. 1-10, 2023c. ISSN 2162-237X, 2182-2388. doi: 10.1109/TNNLS. 2023.3292066.
J Zheng, H Guo, and H Chu. A large scale 12-lead electrocardiogram database for arrhythmia study (version 1.0. 0). PhysioNer 2022Available online orgppttniinneerrggcttteennngg Oaccessed on, 23, 2022.
orgcontentecg arrhythmial0
Jianwei Zheng, Huimin Chu, Daniele Struppa, Jianming Zhang, Sir Magdi Yacoub, Hesham El- Askary, Anthony Chang, Louis Ehwerhemuepha, Islam Abudayych, Alexander Barrett, Guohua Fu, Hai Yao, Dongbo Li, Hangyuan Guo, and Cyril Rakovski. Optimal Multi-Stage Arrhythmia Classification Approach. Scientific Reports, 10(1):2898, February 2020. ISSN 2045-2322. doi:
13

---

Published as a conference paper at ICLR 2025
A DETAIL IN EXPERIMENTAL SETTINGS
A. HYPERPARAMETER SETTINGS
In this section, we provide detailed hyperparameter settings for HeartLang. Due computational resource limitations, the VQ-HBR training and pre-training for the ablation experiments were con- ducted using four NVIDIA GeForce RTX 4090 GPUs, with batch size of 64 per GPU.
Table 4: Hyperparameters for vector-quantized heartbeat reconstruction training.
Hyperparameters
Values
Transformer encoder layers
Transformer decoder layers
Hidden size
768
MI. size
2048
Attention head number
2
Vocabulary size
8192
Collective ECG word dimension
128
Batch size
512
Peak learning rate
5e-5
Minimal learning rate
le-5
Learning rate scheduler
Cosine
Optimizer
Adam W
Adam 3 Weight decay
(0.9.0.99)
le-4
Total epochs
100
Warmup epochs
10
Table 5: Hyperparameters for masked ECG sentence pre-training.
Hyperparameters
Values
Transformer encoder layers
Hidden size
768
MLP size
1024
Attention head number
Batch size
512
Peak learning rate
5e-4
Minimal learning rate
le-5
Learning rate scheduler
Cosine
Optimizer
Adam W
Adam |
(0.9,0.98)
Weight decay
0.05
Total epochs
200
Warmup epochs
5
Gradient clipping
3.0
Mask ratio
0.5
14

---

Published as a conference paper at ICLR 2025
Table 6: Hyperparameters for downstream tine-tuning
Hyperparameters
Values
Batch size
256
Peak learning rate
5e-3
Minimal learning rate
le-5
Learning rate scheduler
Cosine
Optimizer
Adam
Weight decay
0.05
Layer decay
0.9
Total epochs
100
Warmup epochs
10
A.2 DATASET SPLIT
We describe the dataset split in Table For the MIMIC-IV-EG (Gow et al. 2023) dataset, split the training and validation sets with ratio 9:1. The validation set of this dataset is used during the VQ-HBR training stage, but not in the pretraining stage. For the four subsets of PTBXL, adhere to the official split from the official work (Wagner et al. (020). For CPSC2018 (Liu et al.) 2018 and CSN (Zheng al. [2000] (022)) We followed the division method provided by MERL [Liu et al. 2024) to ensure consistency.
Table 7: Details dataset split.
Dataset
Number of Categories
Train
Valid
Test
MIMIC-IV-EGG (Gow et 2023
720,031
80,004
PTBXL-Super WWgnnr et al.
2020)
5
17,084
2,146
2,158
PTBXL-Sub Wagger et al. 2020)
23
17,084
2,146
2,158
PTBXI.-Form WWagner et al.
2020)
19
7.197
901
880
PTBXL-Rhythm (Wagner et al.
2020
12
16,832
2,100
2,098
CPSC2018 (Liu et al. 2018
9
4,950
551
1,376
CSN (Zheng et al. 2020
2022
38
16,546
1,860
4,620
15

---

Published as a conference paper at ICLR 2025
B MORE RESULTS AND DISCUSSIONS
B.1 EVALUATION ON DIFFERENT VOCABULARY SIZES
We present the performance of downstream tasks under different vocabulary sizes in Table g reduced the vocabulary size to 64, similar to the vocabulary size used in previous ECG language processing studies, thereby limiting its semantic expressions. The results show that larger vo- cabulary size leads to significant performance improvements. A larger vocabulary provides richer semantic representations and increases the complexity of pre-training, enabling the model to learn more generalized representations and improve the performance of downstream tasks.
We
Table Linear probing results of two different vocabulary sizes. The best results are bolded, while improved values are marked in green.
PTBXI-Suped
PTBXI-Sub
PTBXI--Form
PTBXI-Rhythn
Vocabulary Size
1%
10%
100%
1
10%
100% %
1%
10% 100%
1%
10%
100%
64
75.89
83.72
86.23
60.97
75.95
86.88
57.10
62.99
75.69
58.41
75.31
88.51
8192
78.94
85.59
87.52
64.68
79.34
88.91
58.70
63.99
80.23
62.08
76.22
90.34
Improvement
3.05
1.87
1.29
3.71
3.39
2.03
1.60
1.00
4.54
3.67
0.91
1.83
B.2 EVALUATION FEWER LEAD CONFIGURATION
Based on the fewer lead configuration recommended by Oh et al. (2022), the configuration is shown in Table and the downstream task validation results are presented in Table [T] In most cases, the downstream performance improves significantly with the increase in the number of leads, especially in the Superclass and Subclass subsets for disease diagnosis. Notably, even under the single-lead condition, the downstream task performance of Heartl surpasses that of most baseline methods Table T This demonstrates that Heartlangg well-adapted to the special case of single-lead configurations, highlighting its strong generalization capability.
Table Fewer lead configuration and selected leads.
Number Leads
Selected Leads
I
2
I, II
3
I, II, V2
6
I, II, III, avR, aVL, aVF
12
I, II, aVR, a aVL, aVE, VI, V2, V3, V4, V5, V6
Table 10: Linear probing results of five different lead combinations. The best results are bolded, with gray indicating the second highest.
PTBXL-Super
PTBXL-Sub
PTEXI-Form
PTBXI-Rhythm
Number of Leads
1%
10%
100%
1%
10%
100%
10%
100%
1%
10%
100%
73.97
79.74
81.02
66.91
77.04
83.11
58.66
66.06
70.98
55.28
74.53
84.53
76.81
83.70
85.14
69.63
79.12
85.89
59.42
68.27
77.16
61.57
81.60
86.98
76.55
84.12
85.97
66.61
78.26
87.68
55,47
67.76
70.46
68.19
83,47
86.27
76.45
83.72
85.66
62.59
77.52
85.99
59.74
68.03
79.46
63.60
83.80
91.44
78.94
85.59
87.52
64.68
79.34
88.91
$8.70
63.99
80.23
62.08
76.22
90.34
16

---

Published as a conference paper at ICLR 2025
B.3 LIMITATIONS AND FUTURE WORKS
Our proposed QRS-Tokenizer relies on the QRS complex features of heartbeats, which are a critical component of its functionality. However, for certain diseases conditions characterized by irregular QRS complexes, the tokenizer may struggle to accurately represent these atypical patterns, leading to performance degradation. Additionally, when segmenting heartbeats, the QRS-Tokenizer pads intervals smaller than 96 with zeros. While this approach is simple and effective for most high- quality datasets, it can partially disrupt the characteristics heartbeats in datasets with significant baseline drift, as their baselines may deviate substantially from zero These challenges highlight the need for future improvements to the QRS-Tokenizer, with focus on enhancing its robustness to handle both irregular QRS complexes and baseline drift effectively, paving the way for more reliable ECG language processing across diverse cardiac conditions.
17

---

Published as a conference paper at ICLR 2025
C MORE VISUALIZATION RESULTS
C.I INDIVIDUAL ECG WORDS AND COLLECTIVE ECG WORDS VISUALIZATION
We further visualized additional individual ECG words and collective ECG words demonstrate
the semantic richness of our constructed ECG vocabulary, as shown in Figure
C.2 ECG SENTENCE VISUALIZATION
We visualize the constructed ECG Sentences, as illustrated in Figures 6 and 8 In these figures, the blue lines correspond the original signals, while the red lines denote the reconstructed signals. As shown in Figure 6, the reconstructed signal demonstrates a notably smoother profile, suggesting that the collective ECG word has effectively captured the morphological characteristies of the heart- beat and exhibits strong generalization capabilities. One notable feature of the QRS- Tokenizee we designed is its ability adaptively segment individual ECG words based heart rate, as demon- strated in Figures 7 and g In Figure 7 due to the relatively fast heart rate, the individual ECG words within the ECG Sentence utilize the full sentence length of 256. In contrast, Figure shows relatively slower heart rate, resulting in the use of only a smaller portion of the ECG Sentence, with the remaining sections zero-filled.
1382
2448
Red CTAO
Figure 5: More ECG Words Visualization.
18

---

Published as a conference paper at ICLR 2025
Figure 6: ECG Sentence Visualization.
19

---

Published as a conference paper at ICLR 2025
Figure 7: Visualization of the ECG Sentence with Fast Heart Rate.
20

---

Published as a conference paper at ICLR 2025
Figure 8: Visualization of the ECG Sentence with Slow Heart Rate.
21


Reviews:
[{"review_id": "GL8l4nJ8k8", "reviewer": "Reviewer_5txW", "summary": "The paper introduces HeartLang, a self-supervised learning framework for ECG language processing that conceptualizes heartbeats as \"words\" and rhythms as \"sentences\" to better capture ECG signal semantics. The framework includes the QRS-Tokenizer, which segments ECG signals into meaningful units, and, a transformer-based model for learning spatiotemporal and semantic representations. The author's proposed approach, which uses heartbeat reconstruction and masked sentence pre-training and conducts experiments on six ECG datasets, outperforming other models in tasks such as rhythm and form classification. This work positions ECG data processing within a language-like framework, aiming to enrich model interpretability and generalizability for clinical applications.", "questions": "It is mentioned that the QRS-Tokenizer is used to segment the raw ECG signals into ECG sentences but as per my understanding the QRS tokenizer should be used to segment into each ECG beats aka words?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper is well-written with good clarity and coherence making the paper easy to read and comprehend. Most work on ECG either focus on beats or on rhythms making it challenging to learn both intra-beat and inter-beat features together which is where this contribution shines. The contribution is significant and interest with the concept of beats as words and rhythms as sentences which would allow learning more comprehensive latent features about the beats itself as well as the relation between the beats similar to human language. The masked training approach although not novel is a novel application with the concept of beats as words and rhythms as sentences and achieved significantly improved performance on popular datasets.", "weaknesses": "It is mentioned that the QRS-Tokenizer is used to segment the raw ECG signals into ECG sentences but as per my understanding the QRS tokenizer should be used to segment into each ECG beats aka words which is confusing. The application of HeartLang at this point is limited and would require investigation on downstream tasks along with comparative studies.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "IAOM7aqhGq", "reviewer": "Reviewer_6RzM", "summary": "This paper introduces HeartLang, a self-supervised learning framework designed for analyzing electrocardiogram (ECG) signals by treating them like a language, with heartbeats as \"words\" and rhythms as \"sentences.\" This approach diverges from conventional ECG analysis by using a language model perspective that preserves the form and rhythm characteristics of ECG signals, enhancing representation learning. Key contributions include:\n\n1. QRS-Tokenizer: This tool segments ECG signals into meaningful \"sentences\" by identifying individual heartbeats (treated as words).\n2. ST-ECGFormer: A spatio-temporal network designed to capture and process the temporal and spatial features of ECG data.\n3. ECG Vocabulary: The largest heartbeat-based vocabulary to date, with 5,394 distinct heartbeat types for diverse cardiac conditions.\n4. Masked ECG Sentence Pre-Training: A pre-training method that learns rhythm-level representations by masking parts of ECG \"sentences.\"\n\nHeartLang was tested on three public datasets, showing improved performance over other ECG self-supervised learning methods in areas such as general representation learning and downstream tasks, such as classification. This approach opens up a language-based approach for ECG research.", "questions": "The approach of this research, which treats each heartbeat as a word and the rhythm of the time series as a sentence, is a very good idea for bringing the success of large-scale language models in natural language to electrocardiograms, and as described in this paper, it has also been successfully implemented. Continuing from the above weakness section, it would be even better if there was a clear discussion of the limitations in the main text.", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This study on the HeartLang self-supervised learning framework has several strengths that make it a valuable contribution to the field of ECG analysis:\n- Innovative ECG Language Perspective: The approach of treating ECG signals as a language, where heartbeats are \"words\" and rhythms are \"sentences,\" is novel. This shift from conventional time-series analysis captures the unique form and rhythm characteristics of ECG signals, which can be critical for diagnosing cardiac conditions.\n- Introduction of the QRS-Tokenizer: This component is specially designed to segment ECG signals into semantically meaningful units, allowing the model to focus on clinically relevant patterns in the data. The QRS-Tokenizer aligns the structure of the data with natural language processing (NLP) techniques, enhancing the depth and utility of the learned representations.\n- Transformer-Based Architecture with ST-ECGFormer: The framework’s backbone, ST-ECGFormer, is a transformer-based model specifically designed to capture spatio-temporal features in ECG data. This model helps HeartLang learn complex temporal and spatial dependencies in the signals, enhancing the quality of the extracted features for further tasks.\n- Flexibility in Downstream Applications: HeartLang’s pre-trained representations can be adapted for various downstream tasks, such as classification, providing a flexible tool for researchers and clinicians to build specific diagnostic models based on general ECG representations.\n\nThese strengths make HeartLang a promising advancement for ECG data analysis, particularly in enhancing the interpretability and applicability of ECG-based machine learning models in clinical diagnostics.", "weaknesses": "This approach would be effective in cases where the QRS waveform is recorded on an electrocardiogram, i.e. in cases of ST abnormalities due to ischemic heart disease or bundle branch block. However, many heart diseases cause significant irregularities in the heart rhythm itself, and these conditions are generally more directly linked to life. It is difficult to think of the HeartLang approach as being suitable for these conditions, and it is thought that it will be necessary to supplement it with other approaches in order to cover the whole of electrocardiography or heart disease.\nThe scope of this study may be narrow than the general interest of ICLR main conference.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "gIP4yOmZ1i", "reviewer": "Reviewer_HUR2", "summary": "This work addresses limitations in deep learning for ECG analysis (usually based on fixed-size and fixed-step time windows) by proposing a novel approach that treats ECG signals like language, where heartbeats are \"words\" and rhythms are \"sentences\". The authors introduce the QRS-Tokenizer to segment ECG signals into meaningful \"sentences\" and present HeartLang, a self-supervised framework that learns ECG representations at the form and rhythm levels. A comprehensive heartbeat-based ECG vocabulary is provided. This approach shows strong performance across 6 ECG data sets.", "questions": "- To what extent does the current manuscript differ to the manuscript accepted at KDD-AIDSH 2024? What are the novel findings?\n\n-> Has been clarified in the revisions, thank you!", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- Original idea to treat ECG signal as language in form of sentences and words\n- Straightforward approach to tokenize the ECG signal into words and sentences (QRS-Tokenizer)\n- Large ECG vocabulary - also useful and interpretable from a clinical perspective\n- Systematic comparison of fixed-size and fixed-step time windows vs. language approach\n- Benchmarking of results (using standard settings of MERL)\n- Informative ablation study regarding spatial / temporal embeddings, pretraining and vocabulary set", "weaknesses": "- A similar version of this manuscript has already been accepted: https://openreview.net/pdf?id=sdczcOS3n9 (at KDD-AIDSH 2024)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "73dFLxZS7B", "reviewer": "Reviewer_xUd7", "summary": "This paper proposes a self-supervised learning framework for ECG language processing considering ECG beats as words and sequence of of beats as sentences mimicking learning ECG representation at form and rhythm levels. Instead of segmenting ECG signal into fixed-size and fixed-step windows view, as done by other contemporary studies, in this study, ECG beats (area consisting of PQSRT waves) are isolated by locating R-peaks and area around it and repeating it to retrieve sequence of beats for analysis. Transformer-based backbone architecture was used as encoder named as ST-ECGFormer to input ECG sentences which consists of ECG word (token) embedding and spatio-temporal and position embedding. Using vector quantisation, ECG vocabulary was auto generated in order to identify and cross-match similar ECG beats (words). A reconstruction training was adopted by masking parts of ECG words and using encoder/decoder architecture to predict the collective ECG word indices of the masked parts based on the unmasked individual ECG words. Authors have done experimentation on 3 publicly available ECG datasets and the proposed method achieved superior performance on heartbeat class and rhythm class classification downstream tasks after pre-training followed by fine-tuning which shows superior performance in several cases. ECG vocabulary was created and visualisation shows ECG words with the same index exhibit similar semantic information in terms of heartbeat representation.", "questions": "Overall, the paper could be a moderate ECG representation learning contribution, with some practical and experimental issues requiring clarification. Given these clarifications in an author response, I would be willing to increase the score.\n\nThe ECG vocabulary was learned which shows similar semantic information, however, its consistency, as well as its randomness needs to be rigorously explored. It's necessity seems questionable due to the fact that normal heartbeats from a single recording may have turned into separate ECG words in the vocabulary which does not make much sense. Although the methodological process can generate it, its utility needs to be justified first. This thus undermines one of the claimed novelty.  An analysis of vocabulary stability across different recordings or subjects seems necessary.\n\n12-lead ECGs were considered as data where some leads can be directly deduced from lead II and III including I, aVR, aVL, and aVF - such channel redundancies may have impact on ECG vocabulary generation. The framework does not show generalisability for less number of channels or even a single channel ECG where there is no spatial items. Experiments with reduced lead sets or single-channel ECGs would be beneficial that would demonstrate the framework's generalizability.\n\nData split lacks clarity if the train, test and validation splits combine segments across recordings or subject wise splits were considered. A clarification of data splitting strategy in the methodology seems necessary. This is because, if subject wise splits were not considered, then there will be data leakage.\n\nThe proposed framework is a variant of EEG based framework, however, the contribution is apparently for ECG modality and the data slicing philosophy. Authors should clarify the claim of framework based novelty accordingly.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "A self-supervised framework was proposed for ECG representation learning considering an ECG signals as a sequence of beats, thus slicing them to include each of them in order to learn their form and a sequence of them to learn rhythms. A ECG vocabulary was created  which shows similar semantic information in terms of heartbeat representation.", "weaknesses": "The study lacks novelty in the framework itself since it is based on an existing one published in past ICLR venue, titled \"LARGE BRAIN MODEL FOR LEARNING GENERIC REPRESENTATIONS WITH TREMENDOUS EEG DATA IN  BCI\". However, the novelty lies in considering another modality of data, ECG. The way ECG beats are curated to formulate the problem in this study is interesting. \n\n- The performance difference compared to other self-supervised methods in the literature were discussed, however, it is unclear if the difference is due to the proposed framework or slicing ECG precisely to consider each beat in forming a sequence of beats, or just for considering ECG segments fixed-size and fixed-step. Applying the beat slicing mechanism used in this study to another study where fixed-size and fixed-step windowing were used would help.\n\n- The generated ECG vocabulary is interesting to see, however, its motivation is unclear considering the fact that normal beats of a single recording may yield to show variance in the vocabulary without any clinical significance. It is not shown in this study how diverse these beats are for each category such as how many words the model comes up with for normal beats of a single recording. A domain specific discussion seems necessary to understand the utility of ECG vocabulary.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)