Reading Your Heart: Learning ECG Words and Sentences via Pre-training ECG Language Model

Authors: jiarui  jin
haoyu  wang
jun  li
sichao  huang
jiahui  pan
shenda  hong
Venue: KDD_org_2024_Workshop_AIDSH
Published: 2024-06-04 18:28:45

Abstract: Electrocardiograms (ECGs) are essential for the clinical diagnosis of arrhythmias and other heart diseases, but deep learning methods based on ECGs often face limitations due to the need for high-quality annotations. Although previous ECG self-supervised learning (eSSL) methods have made significant progress, they typically treat ECG signals as general time-series data, using fixed steps and window sizes, which often ignore the heartbeat and rhythmic characteristics and potential semantic relationships in ECG signals. In this work, we introduce a novel perspective on ECG signals, treating heartbeats as words and rhythms as sentences. Based on this perspective, we propose HeartLang, a novel self-supervised learning framework for ECG language processing. Within this framework, we construct an ECG vocabulary and pre-train the model using masked prediction on ECG sentences to learn both heartbeat-level and rhythm-level representations, uncovering the latent semantic relationships in ECG signals. We also developed three parameter scales for HeartLang, namely, HeartLang-Small, HeartLang-Base, and HeartLang-Large, and conducted pre-training and downstream task testing on the standard benchmark dataset PTB-XL.The experimental results demonstrate that our method exhibits superior performance compared to other eSSL methods.

Main paper: 
 introduction: 
 Electrocardiogram (ECG) is a common type of clinical data used to monitor cardiac activity, and is frequently employed in diagnosing cardiac diseases or conditions impairing myocardial function [12] . A primary limitation of using supervised deep learning methods for ECG signal analysis is their dependency on large-scale, expertreviewed, annotated high-quality data. Moreover, even with sufficient data, these methods are often designed to address specific tasks, which curtails the generalization ability of the model. To overcome these challenges, ECG self-supervised learning (eSSL) has demonstrated significant efficacy by training on vast amounts of unlabeled cardiac recordings to derive generic ECG signal representations, which are then fine-tuned for specific downstream tasks [16] [5] [9] .Current eSSL methods can be primarily classified into two categories: contrastive-based methods and reconstruction-based methods. The core principle of contrastive-based methods involves creating positive and negative sample pairs, with the objective of maximizing the similarity of positive pairs and minimizing the similarity of negative pairs in the feature space. Reconstruction-based methods focus on training a model to reconstruct the original input from partial or transformed data, thereby learning effective data representations. However, almost all methods treat ECG signals as ordinary time series data. These methods have two significant drawbacks:Ignoring Heartbeat and Rhythmic Characteristics of ECG. In ECG diagnostics, heartbeat characteristics are essential. For example, myocardial infarction is diagnosed by observing ST segment elevation [15] . Likewise, cardiac rhythm characteristics are critical, as arrhythmias are identified based on the overall cardiac rhythm [1] . However, existing eSSL methods typically employ fixed-size and fixed-step time windows to segment the data. This method treats ECG signals as ordinary time series signals, thus neglecting the unique heartbeat and rhythmic features inherent in ECG signals.Ignoring Latent Semantic Relationships of ECG. Due to significant differences in heart rate and other factors between different subjects, and even among different samples from the same subject, using fixed-size and fixed-step time windows to segment data leads to substantial discrepancies among samples. The learning of latent semantic representations depends on the similarity between samples, but the variability introduced by fixed-size and fixed-step time windows disrupts this similarity. Different samples are not processed within a unified feature space, resulting in inconsistent feature representations. This lack of uniformity hinders the model's ability to find effective similarities when learning latent semantic representations, leading to suboptimal learning outcomes.To address these challenges, we propose a novel self-supervised learning framework for ECG language processing (ELP) [13] named HeartLang. A significant distinction between ECG and other physiological signals is the clarity of heart rate over time, where most heartbeats can be distinctly recognized and differentiated. The core idea of this framework is to treat heartbeats as words and rhythms as sentences. First, we segment the ECG signal using QRS complexes to obtain the original heartbeat intervals. Then, we concatenate the heartbeat intervals from different leads to form ECG sentences. To overcome inter-subject variability, we do not directly treat the original heartbeat intervals as ECG words. Instead, we construct a large ECG vocabulary, quantifying heartbeats from different subjects into discrete ECG sentence embeddings, and use these embeddings to reconstruct the original heartbeat intervals to learn representations of the same heartbeat across different subjects. ECG vocabulary provides a compact, general, and meaningful representation of different heartbeat intervals. To enable the framework to learn representations of the overall ECG rhythm, we mask parts of the ECG sentences and predict the missing ECG words during large-scale pre-training. Through these approaches, our method can learn both heartbeat-level and rhythm-level representations of ECG signals without labels, and extract latent semantic representations in ECG sentences using ECG vocabulary and masked prediction pre-training. The main contributions of this work are summarized in below:• We propose HeartLang, a novel self-supervised learning framework for ECG language processing, designed to learn information-rich representations at different levels from unlabeled ECG signals.  
methods: 
 In this section, we provide a detailed explanation of the specific structure of the HeartLang framework. We first define multi-lead ECG data as 𝑋 ∈ R 𝐶 ×𝑇 , where 𝐶 represents the number of ECG leads (electrodes) and 𝑇 represents the total timestamps. The configuration of ECG leads follows the standard 12-lead ECG setup. The overview of the framework is shown in Figure 2 . The use of the framework can be divided into three steps. First, constructing the ECG vocabulary is achieved through the steps in the Section 2.3. Second, masked pre-training of the framework is performed as described in the Section 2.4. Finally, fine-tuning for downstream tasks is carried out, which can be divided into linear probing, where only the classification head is trained in the ECGFormer, and full parameter fine-tuning, where all parameters of the ECGFormer are trained.A key concept of our method is to treat heartbeats as words, thus making the segmentation of the original ECG signal into heartbeat sequence essential. We introduce QRSTokenizer, a tokenizer that segments ECG signals into heartbeat sequences based on QRS waveforms. Initially, we bandpass filter the I-lead signal between 5 and 20 Hz, apply moving wave integration (MWI) with a Ricker wavelet to the filtered signal, and save the squared integration signal. We then traverse the local maxima of the MWI signal; each local maximum that meets the criteria of being post-refractory period and exceeding the QRS detection threshold is classified as a QRS complex. Following detection, we obtain the indices of the detected QRS complexes 𝑄 = {𝑞 𝑖 |𝑖 = 1, . . . , 𝑁 }, where 𝑁 denotes the number of detected QRS indices per sample, which varies between samples. Assuming the time window size is 𝑡. For each lead, we center each index in 𝑄, using the midpoint between every two indices as the interval boundaries, and independently segment the QRS complex intervals for each lead. If the segmented region is smaller than 𝑡, we pad it with zeros to match the required size. After segmentation, we concatenate the heartbeat intervals of the 12 leads in sequence, forming the overall ECG sentence 𝑥 ∈ R 𝑙 ×𝑡 , where 𝑙 represents the sequence length and 𝑡 denotes the time window size. Given the variability in heart rates across samples, the resulting sequence lengths are inconsistent. Similar to natural language processing, we set 𝑙 to the maximum length of the ECG sentence. If the length of the ECG sentence is less than 𝑙, it will be padded to 𝑙 through the zero interval; if the length of the ECG sentence exceeds 𝑙, the interval length will be truncated to 𝑙.To construct an ECG vocabulary and predict these vocabularies during pre-training, we designed a backbone network capable of extracting and reconstructing the semantic representations contained in ECG sentences. This backbone network is employed in Embedding. ECG signals have high temporal resolution, and the QRS complexes forming ECG sentences contain rich temporal features. By mapping these QRS complexes into a new feature space, we can more effectively extract and encode these features. We use a 1-D convolutional layer-based mapper to convert heartbeats in ECG sentences into tokens. After mapping, each heartbeat interval in the ECG sentence can be represented as 𝑒 = 𝑒 𝑖,𝑗 ∈ R 𝐷 | 𝑖 = 0, 1, . . . , 𝐶, 𝑗 = 0, 1, . . . ,𝑇 , where 𝐷 is the dimension of the elevated feature space.Here, 𝑒 0,0 represents the padded heartbeat interval, which will be explained in detail in section 2.2.2. The mapped ECG sentence is represented as: 𝑋 ∈ R 𝑙 ×𝐷 .Embedding. In our designed ECG sentences, heartbeat intervals have two inherent characteristics: different leads and varied time positions. For example, in one sample, the centers of the first two QRS complexes are within the first second, while in other samples, only one QRS complex center falls within this interval.To enable the ECGFormer to better capture the temporal and spatial information of heartbeat intervals in ECG sentences and to understand the differences between various heartbeat intervals, we initialized a temporal embedding list 𝑇 𝐿 = {𝑡 0 , 𝑡 1 , 𝑡 2 , . . . , 𝑡 𝑇 } and a spatial embedding list 𝑆𝐿 = {𝑠 0 , 𝑠 1 , 𝑠 2 , . . . , 𝑠 𝐶 }, both of which are 𝐷-dimensional and learnable during the training process.As described in section 2.1, each heartbeat interval can be identified by its lead and its time window. For the time window of each heartbeat interval, we divided the original data into 10-second segments. The specific time window of each heartbeat interval depends on the peak of the QRS complex within that interval. However, due to zero-padding applied to align the ECG sentence to the maximum length, we need additional identification. In this paper, we set both the lead and time window of zero-padded intervals to 0. This approach simplifies and extends the process of adding temporal and spatial embeddings, as it does not consider the number of leads or time windows.Temporal embedding 𝑡 0 and spatial embedding 𝑠 0 are used to distinguish whether data within a heartbeat interval is zero-padded; only when an interval is zero-padded do we add the 𝑡 0 temporal embedding and the 𝑠 0 spatial embedding to it. In other cases, we add temporal and spatial embeddings based on the lead and the time window the heartbeat interval belongs to. Thus, any heartbeat interval 𝑒 𝑖,𝑗 from the mapper, after adding temporal and spatial embeddings, is represented as:𝑒 = {𝑒 𝑖,𝑗 + 𝑠 𝑖 + 𝑡 𝑗 |𝑖 = 0, 1, . . . , 𝐶, 𝑗 = 0, 1, . . . ,𝑇 }Here, the temporal and spatial embeddings serve as absolute position encodings.In addition to using temporal and spatial embeddings to capture the absolute positional relationships within ECG sentences, we also introduced Rotary Position Embedding (ROPE) to enhance the ECGFormer's understanding of relative positional relationships. This dual encoding approach makes the ECGFormer more flexible and comprehensive in understanding ECG sentences. Assume the input vector is 𝑋 ∈ R 𝑙 ×𝐷 , with dimension 𝐷 and sequence length 𝑙. ROPE incorporates positional information directly into the features of heartbeat intervals by applying a rotational transformation to the position encoding. The feature representation after applying ROPE, denoted as ROPE(𝑋 ), is calculated as follows: First, define an angle matrix Θ, where each element 𝜃 𝑖,𝑗 represents the angle for position 𝑖 and dimension 𝑗:𝜃 𝑖,𝑗 = 𝑖 10000 2𝑗 𝐷for 𝑖 = 0, 1, 2, . . . , 𝑙 -1 and 𝑗 = 0, 1, 2, . . . , 𝐷 2 -1. Construct a rotation matrix 𝑅(𝜃 ) to apply the rotational transformation to the input vectors:𝑅(𝜃 ) =cos(𝜃 )sin(𝜃 ) sin(𝜃 ) cos(𝜃 ) Split the input vector 𝑋 into two sub-vectors 𝑋 even and 𝑋 odd :EQUATION)EQUATION)Finally, for each position 𝑖 and dimension 𝑗, apply the rotation matrix 𝑅(𝜃 𝑖,𝑗 ) to transform the vectors:ROPE(𝑋 ) 𝑖,2𝑗 = 𝑋 𝑖,2𝑗 cos(𝜃 𝑖,𝑗 ) + 𝑋 𝑖,2𝑗+1 sin(𝜃 𝑖,𝑗 ) ROPE(𝑋 ) 𝑖,2𝑗+1 = -𝑋 𝑖,2𝑗 sin(𝜃 𝑖,𝑗 ) + 𝑋 𝑖,2𝑗+1 cos(𝜃 𝑖,𝑗 )The new feature representation after rotation, denoted as ROPE(𝑋 ), is:EQUATION)The final feature representation encodes each position 𝑖 and dimension index 𝑗 in the entire input sequence 𝑋 , meaning that each element ROPE(𝑋 ) 𝑖,𝑘 contains a combination of the original feature 𝑋 and positional information, where 𝑘 = 0, 1, . . . , 𝐷 -1. where x is the reconstructed ECG sentence and 𝑓 𝑑 is the decoder.To make the update of the ECG vocabulary more stable, we adopt an exponential moving average (EMA) strategy. The mean squared error (MSE) loss is utilized to guide the reconstruction. Finally, the loss function for training the vector-quantized heartbeat L 𝑉 = ∑︁ 𝑥 ∈ D 𝑙 ∑︁ 𝑖=1 ∥ x𝑖 -𝑥 ∥ 2 2 + 𝑠𝑔 ℓ 2 (𝑝 𝑖 ) -ℓ 2 𝑣 𝑧 𝑖 2 2 + ℓ 2 (𝑝 𝑖 ) -𝑠𝑔 ℓ 2 𝑣 𝑧 𝑖 2 2where 𝐷 represents all the ECG sample data, and 𝑠𝑔 denotes the stop-gradient operation, which is defined as the identity function in the forward pass and has zero gradient.In this section, we introduce the pre-training part of the Heart-Lang framework, as depicted in the lower half of figure 2 . This is a focal point of the entire HeartLang framework, aimed at utilizing unmasked parts of heartbeats and ECG Vocabulary containing heartbeat semantic information within ECG sentences. This forces the ECGFormer to learn rhythm-level features of ECG sentences, which inherently possess generalizability, thus effectively applying the rhythm-level knowledge learned by the ECGFormer to various downstream tasks. The pre-training component primarily includes ECG heartbeat masking and ECG sentence masking prediction.To enable the ECGFormer to learn the rhythm-level features of ECG sentences, we perform random masking on the entire ECG heartbeat sentence, allowing the ECG-Former to understand the content of the entire ECG sentence based on the unmasked heartbeats. For the heartbeat sentence 𝑋 ∈ R 𝑙 ×𝐷 obtained through the QRS Tokenizer, where each vector can be represented as:𝑒 = {𝑒 𝑖 |𝑖 = 1, . . . , 𝑙 }We randomly generate a mask 𝑀 = {𝑚 𝑖 |𝑖 = 1, . . . , 𝑙 } where 𝑚 𝑖 ∈ {0, 1}, setting the masking rate to 0.5, meaning that half of the heartbeats in each ECG heartbeat sentence are masked. We use a learnable mask token 𝑒 𝑀 ∈ R 𝑑 to replace the masked heartbeats. Thus, the entire heartbeat sentence can be represented as:𝑋 = {𝑒 𝑖 : 𝑚 𝑖 = 0|𝑖 = 1, . . . , 𝑁 } ∪ {𝑒 𝑀 : 𝑚 𝑖 = 1|𝑖 = 1, . . . , 𝑁 }It is then enhanced with temporal and spatial position encoding, before being input to the Transformer's Encoder.Vocabularies. ECG sentences are first vector quantized as described in section 2.4.1, obtaining the indices of each word in the ECG sentence. Subsequently, we extract the indices of the words corresponding to the masked parts of the ECG sentence to achieve a one-to-one correspondence between ECG words and masks. The task of this stage is to predict the word indices of the masked parts based on the unmasked parts of the ECG sentence, by minimizing the discrepancy between the predicted word indices and the true word indices, thus facilitating model training. Therefore, the pre-training part uses the information in the ECG sentence itself as labels, with the ECG sentence as the input, representing a typical form of self-supervised learning. The target loss function L 𝑃 = - ∑︁ 𝑥 ∈ D ∑︁ 𝑚 𝑖 =1 log 𝑝 (𝑣 𝑖 |𝑋 )3 EXPERIMENTS AND RESULTS 
dataset: 
 The PTBXL dataset is utilized for pre-training and fine-tuning downstream tasks. This dataset comprises 21,837 ECG signals collected from 18,885 patients. Each sample consists of a 12-lead ECG recorded for 10 seconds at a sampling rate of 500 Hz. The dataset is annotated according to SCP-ECG and includes four multi-label classification subsets: Diagnostic superclass (5 classes), Diagnostic subclass (23 classes), Form (19 classes), and Rhythm (12 classes). Each downstream task category contains a varying number of samples. The official data split is followed for training, validation, and testing.All samples in the PTBXL dataset are first downsampled to 100 Hz to enhance computational efficiency. The construction of the ECG vocabulary and the pre-training of HeartLang are performed using all samples from the PTBXL dataset. Notably, no labels are required for training in these two tasks. We propose three variants of HeartLang with different parameter sizes: HeartLang-Small, HeartLang-Base, and HeartLang-Large. The specific parameter configurations for each model are detailed in the appendix A. In downstream finetuning tasks, two evaluation methods are proposed: full-parameter fine-tuning and linear probing. For full-parameter fine-tuning, none of the layers in ECGFormer are frozen, allowing all layers to be trained. For linear probing, the ECGFormer encoder is kept frozen, and only the parameters of the newly initialized linear classifier are updated. Each task is tested using 1%, 10%, and 100% of the training data to evaluate the generalization capability of HeartLang in few-shot scenarios. For all downstream tasks, macro AUC is used as the evaluation metric.We first evaluate our method using linear probing, as presented in Table 1 . It is noteworthy that the comparative methods' results mainly come from MERL, one of whose major contributions is providing a benchmark from many eSSL methods. It must be stated that the results in MERL were pre-trained on the MIMIC-IV dataset, while in our work, due to computational resource limitations, the HeartLang were only pre-trained on PTB-XL. The sample size of MIMIC-IV is about thirty times larger than that of PTB-XL. Finally, MERL is not an eSSL method but a multimodal method incorporating text, whereas the other comparative methods are all eSSL methods. The results show that our method demonstrates strong competitiveness in linear probing. Specifically, in the four downstream tasks, HeartLang-Base and HeartLang-Small achieved first or second highest in most cases with varying proportions of training data. Particularly, compared to other eSSL methods, our method almost universally outperformed them, except slightly lower performance in the 1% training sample cases of Diagnostic superclass and Diagnostic subclass classification tasks. Compared to MERL, our method achieved comparable or better results in a single modality (i.e., ECG signals alone). Compared to Random Init, where the encoder weights are randomly initialized, our method shows significant improvement. This indicates that our method indeed learns multi-level ECG representations, resulting in competitive performance in downstream tasks. Interestingly, we found that as the parameter size increased, HeartLang-Large did not bring a significant performance boost, which we suspect is due to the limited sample size of our pre-training dataset.Table 2 presents the performance results of different fine-tuning strategies. "Linear" represents linear probing, which only trains the final linear classification head while freezing the encoder. "All" represents full parameter fine-tuning, where all layers of the model are trained. The results show that, in most cases across the four downstream datasets, the full parameter fine-tuning strategy outperforms the linear probing strategy. Only in rare cases, such as HeartLang-Small on PTBXL-Rhythm and HeartLang-Base on PTBXL-Form, does the performance slightly fall behind the linear probing strategy. Additionally, most of the best results are concentrated in the HeartLang-Small model. We speculate that this is because the downstream tasks have relatively small amounts of data, and using models with very large number of parameters may lead to overfitting, thus reducing performance.We selected 256 samples for visual analysis of the ECG vocabulary, with specific results shown in Appendix B. To more prominently display the original forms of ECG words in different ECG vocabularies, we visualized the top 20 most frequently occurring ECG words, with the results shown in Figure 3 . The distinct clusters observed in the visualization suggest that there is a natural grouping of ECG words based on their semantic or morphological similarities. Furthermore, the proximity of points to one another can provide insights into the relationships between different ECG words. ECG words that are closely located may share similar characteristics and may jointly influence the occurrence of different diseases. 
conclusion and future work: 
 In this paper, we introduce HeartLang, a novel self-supervised framework for ECG language processing. This framework learns heartbeat-level and rhythm-level representations by constructing an ECG vocabulary and pre-training by masked prediction on ECG sentences, uncovering the latent semantic relationships in ECG signals. We provide a novel perspective on ECG signals by viewing heartbeats as words and rhythms as sentences, and use this approach to conduct large-scale pre-training on a vast amount of unlabeled ECG signals. Extensive experiments are conducted on benchmark datasets with varying proportions of training data, evaluating the performance of HeartLang under three parameter scales. The results show that our method exhibits significant advantages over other comparative eSSL methods across different scenarios.In the future, we will further explore the effectiveness of the pretraining framework on larger-scale datasets. We hope our work provides an effective solution for future ECG foundational models and benefits the entire ECG representation learning research community.  
 
 Appendix:  



Reviews: review: title: title: Official Review
main_review: main_review: Summary:
This paper introduces a novel self-supervised learning framework called HeartLang for ECG language processing. It highlights the understanding of ECG signals by treating heartbeats as words and rhythms as sentences. By constructing an ECG vocabulary and taking advantage of masked prediction during pre-training, HeartLang learns both heartbeat-level and rhythm-level representations, uncovering latent semantic relationships in ECG signals. The framework offers three scales (HeartLang-Small, HeartLang-Base, and HeartLang-Large) which outperforms other eSSL methods in experiments. This innovative approach paves the way for more effective ECG foundational models and benefits the ECG representation learning field.

Strengths:
1. The paper is well-written and organized.
2. The paper introduces an innovative perspective on ECG signals by viewing heartbeats as words and rhythms as sentences, and
extract latent semantic representations in ECG signals using self-supervised learning.
3. The proposed framework is simple yet effective, which is shown in the experiments.

Weaknesses:
1. The datasets used in the paper is not large enough for evaluating the scalability.
2. The method proposed enables training of all layers, but what about its efficiency? The experiments include baseline models that only fine-tune linear layers. Is Lora utilized in this fine-tuning process? The experimental section lacks detailed information on the techniques employed.
title: title: Reading Your Heart: Learning ECG Words and Sentences via Pre-training ECG Language Model
main_review: main_review: Pre-training ECG as an NLP task is a very interesting approach. By converting complex time series pre-training methods to well-explored NLP pre-training algorithms, researchers are often able to gain more insights. Some well-tested NLP algorithms can also be applied to the ECG domain, such as BERT. I believe that this work, as well as subsequent efforts, will contribute to the community. At the same time, we are eager to see the creation of a more comprehensive vocabulary and the resolution of heartbeat signal encoding in some extreme cases in upcoming works.

Strength:

- Using a tokenizer can effectively reduce the computational load during the pre-training process. Compared to directly computing time series, computing sentences is more efficient.
- The author converts complex ECG signals into words and sentences, and trains the ECG signals using a BERT-like pre-training method. This approach integrates knowledge from more mature fields into the realm of ECG signal processing.

Weakness:

- The QRSTokenizer used in this paper seems to have lost some key features when encoding ECG waveforms, such as heart rate. Many conditions of heart diseases are characterized by abnormal waveforms under high heart rate states. After Tokenization, the length of a sentence to some extent represents its heart rate, but truncation will lead to the loss of crucial information.

Question:
- Unlike NLP, ECG signals do not have clear BOS and EOS; there is inevitably some information loss at the beginning and end of the signals. For example, if a signal is missing a Q wave due to collection issues, how should these incomplete 'words' be handled?
- Some abnormal heartbeats do not have clear QRS waves, such as ventricular beat and cardiac arrest. Even some ECG equipment may pick up noise. Can the ECG vocabulary cover these situations?
- In Section 2.3, Figure X is missing. Also, I do not quite understand the purpose of Section 2.3; why is it necessary to use reconstruction loss for trying word embeddings? Is this loss one of the optimization targets during the pre-training process, or is this loss function only used for creating the vocabulary?