Reading Your Heart: Learning ECG Words and Sentences via Pre-training ECG Language Model

Authors: Jiarui Jin, Haoyu Wang, Hongyan Li, Jun Li, Jiahui Pan, Shenda Hong
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords: Electrocardiogram, ECG, Cardiac signal, Self-supervised learning, ECG language processing

Abstract:
Electrocardiogram (ECG) is essential for the clinical diagnosis of arrhythmias and other heart diseases, but deep learning methods based on ECG often face limitations due to the need for high-quality annotations. Although previous ECG self-supervised learning (eSSL) methods have made significant progress in representation learning from unannotated ECG data, they typically treat ECG signals as ordinary time-series data, segmenting the signals using fixed-size and fixed-step time windows, which often ignore the form and rhythm characteristics and latent semantic relationships in ECG signals. In this work, we introduce a novel perspective on ECG signals, treating heartbeats as words and rhythms as sentences. Based on this perspective, we first designed the QRS-Tokenizer, which generates semantically meaningful ECG sentences from the raw ECG signals. Building on these, we then propose HeartLang, a novel self-supervised learning framework for ECG language processing, learning general representations at form and rhythm levels. Additionally, we construct the largest heartbeat-based ECG vocabulary to date, which will further advance the development of ECG language processing. We evaluated HeartLang across six public ECG datasets, where it demonstrated robust competitiveness against other eSSL methods. Our data and code are publicly available at https://github.com/PKUDigitalHealth/HeartLang.

Decision: accept (poster)
Reviewer scores: 6, 8, 8, 6

Reviews and discussion:
<thread>
creation date:
1729945766465

thread_type:
official_review

<thread_object>
creator: reviewer_xud7

<content>
summary:
This paper proposes a self-supervised learning framework for ECG language processing considering ECG beats as words and sequence of of beats as sentences mimicking learning ECG representation at form and rhythm levels. Instead of segmenting ECG signal into fixed-size and fixed-step windows view, as done by other contemporary studies, in this study, ECG beats (area consisting of PQSRT waves) are isolated by locating R-peaks and area around it and repeating it to retrieve sequence of beats for analysis. Transformer-based backbone architecture was used as encoder named as ST-ECGFormer to input ECG sentences which consists of ECG word (token) embedding and spatio-temporal and position embedding. Using vector quantisation, ECG vocabulary was auto generated in order to identify and cross-match similar ECG beats (words). A reconstruction training was adopted by masking parts of ECG words and using encoder/decoder architecture to predict the collective ECG word indices of the masked parts based on the unmasked individual ECG words. Authors have done experimentation on 3 publicly available ECG datasets and the proposed method achieved superior performance on heartbeat class and rhythm class classification downstream tasks after pre-training followed by fine-tuning which shows superior performance in several cases. ECG vocabulary was created and visualisation shows ECG words with the same index exhibit similar semantic information in terms of heartbeat representation.

strengths:
A self-supervised framework was proposed for ECG representation learning considering an ECG signals as a sequence of beats, thus slicing them to include each of them in order to learn their form and a sequence of them to learn rhythms. A ECG vocabulary was created  which shows similar semantic information in terms of heartbeat representation.

weaknesses:
The study lacks novelty in the framework itself since it is based on an existing one published in past ICLR venue, titled "LARGE BRAIN MODEL FOR LEARNING GENERIC REPRESENTATIONS WITH TREMENDOUS EEG DATA IN  BCI". However, the novelty lies in considering another modality of data, ECG. The way ECG beats are curated to formulate the problem in this study is interesting. 

- The performance difference compared to other self-supervised methods in the literature were discussed, however, it is unclear if the difference is due to the proposed framework or slicing ECG precisely to consider each beat in forming a sequence of beats, or just for considering ECG segments fixed-size and fixed-step. Applying the beat slicing mechanism used in this study to another study where fixed-size and fixed-step windowing were used would help.

- The generated ECG vocabulary is interesting to see, however, its motivation is unclear considering the fact that normal beats of a single recording may yield to show variance in the vocabulary without any clinical significance. It is not shown in this study how diverse these beats are for each category such as how many words the model comes up with for normal beats of a single recording. A domain specific discussion seems necessary to understand the utility of ECG vocabulary.

questions:
Overall, the paper could be a moderate ECG representation learning contribution, with some practical and experimental issues requiring clarification. Given these clarifications in an author response, I would be willing to increase the score.

The ECG vocabulary was learned which shows similar semantic information, however, its consistency, as well as its randomness needs to be rigorously explored. It's necessity seems questionable due to the fact that normal heartbeats from a single recording may have turned into separate ECG words in the vocabulary which does not make much sense. Although the methodological process can generate it, its utility needs to be justified first. This thus undermines one of the claimed novelty.  An analysis of vocabulary stability across different recordings or subjects seems necessary.

12-lead ECGs were considered as data where some leads can be directly deduced from lead II and III including I, aVR, aVL, and aVF - such channel redundancies may have impact on ECG vocabulary generation. The framework does not show generalisability for less number of channels or even a single channel ECG where there is no spatial items. Experiments with reduced lead sets or single-channel ECGs would be beneficial that would demonstrate the framework's generalizability.

Data split lacks clarity if the train, test and validation splits combine segments across recordings or subject wise splits were considered. A clarification of data splitting strategy in the methodology seems necessary. This is because, if subject wise splits were not considered, then there will be data leakage.

The proposed framework is a variant of EEG based framework, however, the contribution is apparently for ECG modality and the data slicing philosophy. Authors should clarify the claim of framework based novelty accordingly.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730323894151

thread_type:
official_review

<thread_object>
creator: reviewer_hur2

<content>
summary:
This work addresses limitations in deep learning for ECG analysis (usually based on fixed-size and fixed-step time windows) by proposing a novel approach that treats ECG signals like language, where heartbeats are "words" and rhythms are "sentences". The authors introduce the QRS-Tokenizer to segment ECG signals into meaningful "sentences" and present HeartLang, a self-supervised framework that learns ECG representations at the form and rhythm levels. A comprehensive heartbeat-based ECG vocabulary is provided. This approach shows strong performance across 6 ECG data sets.

strengths:
- Original idea to treat ECG signal as language in form of sentences and words
- Straightforward approach to tokenize the ECG signal into words and sentences (QRS-Tokenizer)
- Large ECG vocabulary - also useful and interpretable from a clinical perspective
- Systematic comparison of fixed-size and fixed-step time windows vs. language approach
- Benchmarking of results (using standard settings of MERL)
- Informative ablation study regarding spatial / temporal embeddings, pretraining and vocabulary set

weaknesses:
- A similar version of this manuscript has already been accepted: https://openreview.net/pdf?id=sdczcOS3n9 (at KDD-AIDSH 2024)

questions:
- To what extent does the current manuscript differ to the manuscript accepted at KDD-AIDSH 2024? What are the novel findings?

-> Has been clarified in the revisions, thank you!
</content>
</thread_object>

</thread>
<thread>
creation date:
1730622551803

thread_type:
official_review

<thread_object>
creator: reviewer_6rzm

<content>
summary:
This paper introduces HeartLang, a self-supervised learning framework designed for analyzing electrocardiogram (ECG) signals by treating them like a language, with heartbeats as "words" and rhythms as "sentences." This approach diverges from conventional ECG analysis by using a language model perspective that preserves the form and rhythm characteristics of ECG signals, enhancing representation learning. Key contributions include:

1. QRS-Tokenizer: This tool segments ECG signals into meaningful "sentences" by identifying individual heartbeats (treated as words).
2. ST-ECGFormer: A spatio-temporal network designed to capture and process the temporal and spatial features of ECG data.
3. ECG Vocabulary: The largest heartbeat-based vocabulary to date, with 5,394 distinct heartbeat types for diverse cardiac conditions.
4. Masked ECG Sentence Pre-Training: A pre-training method that learns rhythm-level representations by masking parts of ECG "sentences."

HeartLang was tested on three public datasets, showing improved performance over other ECG self-supervised learning methods in areas such as general representation learning and downstream tasks, such as classification. This approach opens up a language-based approach for ECG research.

strengths:
This study on the HeartLang self-supervised learning framework has several strengths that make it a valuable contribution to the field of ECG analysis:
- Innovative ECG Language Perspective: The approach of treating ECG signals as a language, where heartbeats are "words" and rhythms are "sentences," is novel. This shift from conventional time-series analysis captures the unique form and rhythm characteristics of ECG signals, which can be critical for diagnosing cardiac conditions.
- Introduction of the QRS-Tokenizer: This component is specially designed to segment ECG signals into semantically meaningful units, allowing the model to focus on clinically relevant patterns in the data. The QRS-Tokenizer aligns the structure of the data with natural language processing (NLP) techniques, enhancing the depth and utility of the learned representations.
- Transformer-Based Architecture with ST-ECGFormer: The framework’s backbone, ST-ECGFormer, is a transformer-based model specifically designed to capture spatio-temporal features in ECG data. This model helps HeartLang learn complex temporal and spatial dependencies in the signals, enhancing the quality of the extracted features for further tasks.
- Flexibility in Downstream Applications: HeartLang’s pre-trained representations can be adapted for various downstream tasks, such as classification, providing a flexible tool for researchers and clinicians to build specific diagnostic models based on general ECG representations.

These strengths make HeartLang a promising advancement for ECG data analysis, particularly in enhancing the interpretability and applicability of ECG-based machine learning models in clinical diagnostics.

weaknesses:
This approach would be effective in cases where the QRS waveform is recorded on an electrocardiogram, i.e. in cases of ST abnormalities due to ischemic heart disease or bundle branch block. However, many heart diseases cause significant irregularities in the heart rhythm itself, and these conditions are generally more directly linked to life. It is difficult to think of the HeartLang approach as being suitable for these conditions, and it is thought that it will be necessary to supplement it with other approaches in order to cover the whole of electrocardiography or heart disease.
The scope of this study may be narrow than the general interest of ICLR main conference.

questions:
The approach of this research, which treats each heartbeat as a word and the rhythm of the time series as a sentence, is a very good idea for bringing the success of large-scale language models in natural language to electrocardiograms, and as described in this paper, it has also been successfully implemented. Continuing from the above weakness section, it would be even better if there was a clear discussion of the limitations in the main text.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730691110104

thread_type:
official_review

<thread_object>
creator: reviewer_5txw

<content>
summary:
The paper introduces HeartLang, a self-supervised learning framework for ECG language processing that conceptualizes heartbeats as "words" and rhythms as "sentences" to better capture ECG signal semantics. The framework includes the QRS-Tokenizer, which segments ECG signals into meaningful units, and, a transformer-based model for learning spatiotemporal and semantic representations. The author's proposed approach, which uses heartbeat reconstruction and masked sentence pre-training and conducts experiments on six ECG datasets, outperforming other models in tasks such as rhythm and form classification. This work positions ECG data processing within a language-like framework, aiming to enrich model interpretability and generalizability for clinical applications.

strengths:
The paper is well-written with good clarity and coherence making the paper easy to read and comprehend. Most work on ECG either focus on beats or on rhythms making it challenging to learn both intra-beat and inter-beat features together which is where this contribution shines. The contribution is significant and interest with the concept of beats as words and rhythms as sentences which would allow learning more comprehensive latent features about the beats itself as well as the relation between the beats similar to human language. The masked training approach although not novel is a novel application with the concept of beats as words and rhythms as sentences and achieved significantly improved performance on popular datasets.

weaknesses:
It is mentioned that the QRS-Tokenizer is used to segment the raw ECG signals into ECG sentences but as per my understanding the QRS tokenizer should be used to segment into each ECG beats aka words which is confusing. The application of HeartLang at this point is limited and would require investigation on downstream tasks along with comparative studies.

questions:
It is mentioned that the QRS-Tokenizer is used to segment the raw ECG signals into ECG sentences but as per my understanding the QRS tokenizer should be used to segment into each ECG beats aka words?
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper has a simple three-step architecture to model ECG signals. First, the creation of a QRSTokenizer that uses basic signal processing to identify QRS signals within the ECG data, then an attention based transformer that combines spatial and temporal information. The output of the transformer is then reconstructed using a learned ECG codebook which is then reconstructed into a signal via an ECG codebook. The entire process is trained end to end using a mean squared error loss and a combination of losses to regularize learning the codebook. Overall, the reviewers found the work simple, well motivated and empirically performant. I think the use of the ECG codebook is clever and could be useful in a variety of downstream tasks well beyond its use in the tasks herein. Their finding that "even similar heartbeat morphologies can yield different semantic representations based on contextual information" I believe will be of independent interest to cardiologists studying diseases where such phenomenon are observed clinically.