Guiding Masked Representation Learning to Capture Spatio-Temporal Relationship of Electrocardiogram
Authors: Yeongyeon Na
Minje Park
Yunwon Tae
Sunghoon Joo
Venue: ICLR 2024 poster (2024)
Keywords: Electrocardiogram
ECG
Cardiac signal
Biosignal
Self-supervised learning
Masked auto-encoder
Representation learning
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
TL;DR: We propose ST-MEM, Spatio-Temporal Masked Electrocardiogram Modeling, to learn the general ECG representation, generally applicable to diverse ECG problems by incorporating the spatial and temporal relationship of ECG signal.

Abstract: Electrocardiograms (ECG) are widely employed as a diagnostic tool for monitoring electrical signals originating from a heart. Recent machine learning research efforts have focused on the application of screening various diseases using ECG signals. However, adapting to the application of screening disease is challenging in that labeled ECG data are limited. Achieving general representation through self-supervised learning (SSL) is a well-known approach to overcome the scarcity of labeled data; however, a naive application of SSL to ECG data, without considering the spatial-temporal relationships inherent in ECG signals, may yield suboptimal results. In this paper, we introduce ST-MEM (Spatio-Temporal Masked Electrocardiogram Modeling), designed to learn spatio-temporal features by reconstructing masked 12-lead ECG data. ST-MEM outperforms other SSL baseline methods in various experimental settings for arrhythmia classification tasks. Moreover, we demonstrate that ST-MEM is adaptable to various lead combinations. Through quantitative and qualitative analysis, we show a spatio-temporal relationship within ECG data. Our code is available at https://github.com/bakqui/ST-MEM.

Published as a conference paper at ICLR 2024
GUIDING MASKED REPRESENTATION LEARNING TO CAPTURE SPATIO-TEMPORAL RELATIONSHIP OF ELECTROCARDIOGRAM
Yeongyeon Na", Minje Park", Yunwon Tac", and Sunghoon Joo!
VUNO Inc.
[yeongyeon.ne na, minje. sinje.park, yunwon. tae, sunghoon. joo]@vuno.co
ABSTRACT
Electrocardiograms (ECG) are widely employed as a diagnostic tool for moni- toring electrical signals originating from heart. Recent machine learning re- search efforts have focused on the application of screening various diseases us- ing ECG signals. However, adapting to the application of screening disease is challenging in that labeled ECG data are limited. Achieving general repre- sentation through self-supervised learning (SSL) is well-known approach overcome the scarcity of labeled data; however, a naive application of SSL to ECG data, without considering the spatial-temporal relationships inherent ECG signals, may yield suboptimal results. In this paper, we introduce ST-MEM (Spatio-Temporal Masked Electrocardiogram Modeling), designed to learn spatio- temporal features by reconstructing masked 12-lead ECG data. ST-MEM outper- forms other SSL baseline methods in various experimental settings for arryyth- mia classification tasks. Moreover, demonstrate that ST-MEM is adaptable to various lead combinations. Through quantitative and qualitative analysis, we show spatio-temppral relationship within ECG data. Our code is available /ttps://gitan com haapmm/aaquu//ee STVSSTTMMI MEM
INTRODUCTION
The electrocardiogram (ECG) is non-invasive heart measurement to monitor the electrical activity over time and diagnose diseases. Several supervised learning models have been developed detect various heart diseases through ECG (Siontis et al. 2021). However, since the types of heart disease diverse and the experienced cardiologists who can provide labels are limited, learning the ECG representation for each application (i.e., detecting various heart diseases) is challenging. Recently, supervuper learning (SSL) for general representation has emerged in natural language process- ing (Kenton & Toutanova 2019 Brown et al. 2020) and computer vision (Chen et al., 2020) Grill jet 20200 He 2020 Caron et al.) (021) since can be leveraged for numerous tasks, such translation, sentence classification, image classification, and image generation. In ECG-based diagnosis, there were also similar efforts to learn general representation through SSL to overcome the limited resources and detect various heart discases.
BCG-based representation learning through SSL usually considered in two different learning methods: contrastive and generative learning (Jing & Tian 2020). Contrastive learning (Sarkar & Etemad 2020 Le et al. 2023 Gopal et al. 2021: /Soltanich et al.] 2022] Kiyasseh et al. 2021
Wei et al.
2022) method ensure similarity in the context before and after data augmenta- tion. These data are usually defined by numerous augmentation methods (e.g., cropping. flipping, and shifting); however, despite simple augmentation, the information on the semantic meaning of ECG signals can change drastically (Lan et al. 2023). Generative learning is a method of learning the representation of data by reconstructing all or part of the input, such as a simple framework for masked image modeling (Xie et al., 2022 or Masked Autoencoder (MAE) (He et al.] 2022). In the case of ECG, there are variants MAE that reconstruct input ECG signals (Zhang et 2022
Equal contribution.
Corresponding author.

---

Published as a conference paper at ICLR 2024
Frecial Procial plane/ Limn leads
aVR
AYL
Herizuntal piane
V2 V5
VI
VI
aVF
V3
Figure 1: An illustration of 12-lead electrocardiogram (ECG). ECG signals consist of 12 leads. Each lead is measured from different spatial locations. Limb leads (i.e., I, II, III, aVR, aVI. and aVF) generated from a frontal plane, while precordial leads (i.e., VI, V2, V3, V4, V5, and V6) are obtained from a horizontal plane.
Sawano al. 2022: Hu al., 2023). Generative learning often augmentation-fre or involves simpler data transformations that are more suitable for preserving the integrity of ECG data. This approach ensures that the context and meaningful information in ECG signals are retained.
In ECG-based representation learning, exploit- ing both spatial and temporal information in ECG is significant. For instance, if we have an L-lead ECG, means that cardiac activity is obtained over duration with L views. There- fore, we can understand a heart complemen- tary when its ECG is gained not only spatially or temporally but spatio-temporally. The most common setting is standard 12-lead ECG (i.e., a heart is observed in 12 views) as shown in Figure [ and for some cases, ECG in which its leads are subset of 12-lead (i.e., reduced lead ECG) is acquired.
An illustration of spatio-temporte patchifying. The black dashed box indicates the query patch; each arrow represents the self- attention arrow; each color represents patch, a single input sample for the model. Tempo- ral patchifying from (a) provides three differ- ent patches (i.c.. three different inputs). Spatial patchifying from (b) yields 12 patches for ev- ery 12 leads. Spatio-temporal patchifying from (c) can provide fine-grained input signals for the model, which allows for capturing spatial and temporal relationships.
Figure 2:
In this work, we leverage ECG to learn general representations by introducing simple but ef- fective superviue learning framework us- ing MAE architecture. Throughout this pro- cess, both temporal and spatial information present in the ECG is utilized. The approach involves applying data, as illustrated in Figure 2 (c), with lead indicators such as fead-wise shared decoder, learnable lead embeddings, and separation embedding, as depicted Figure 3 Moreover, we show that our model can be structurally fine-tuned for reduced lead ECG, demonstrating excellent performance only in the standard 12-lead setting but also in limb leads and single leads. Finally, through quanti- tative and qualitative analysis, demonstrated that spatial and temporal features were effectively learned.
patchifying to ECG
Our contributions the following. (1) We propose the simple but effective ECG-specific generative self-supervised learning framework, named ST-MEM Spatio-Temport Masked Electrocardiogram Modeling). (2) ST-MEM can learn general representation by capturing spatio-temporal relationship of ECGs. We show this relationship by quantitative and qualitative analysis. (3) Through extensive experiments, ST-MEM demonstrates comparable performance to other contrastive and generative learning methods, which are widely used in ECG representation learning. We validate that ST- MEM excels both fine-tuning and linear evaluation for the arrhythmia classification task. This robust performance extends across various scenarios, such as scarcity of labeled data and reduced lead settings.
BACKGROUNDS: ECG
The ECG is a non-invasive heart measurement to observe the electrical signals over time and di- agnose diseases. A standard 12-lead ECG, interpreted multivariate time series, is the most
2

---

Published as a conference paper at ICLR 2024
Plodlt
Puck
Shared Decoder (Lead1)
Shared Decoder (Lead-2)
Shared Decoder (Lead
PMles
Piap
Pilue
Trooded
Atusta
Span
Fund
Makid
Spin
exbediigg
Prn
Pwy
Pro
Pinur
Prn:
Ponsi
retelling
Lcvl
Lecal
Lesd
Lese
Seaal
Leaf
Lest"
rasdon punchar
Favi
Putidrag
Paon
inton
Passh
Pall
Pack?
Figure 3: An overview of our proposed method. ST-MEM consists of encoder and decoder for reconstructing the masked ECG signals. The encoder takes patchfied ECG signals with lead and position embedding. The shared decoder reconstructs the masked ECG signals for each lead by utilizing the encoded representations.
common measurement setting that provides spatial and temporal information regarding the heart. As shown in Figure | the 12 leads are I, II, III, aVR, aVL, aVF, VI, V2, V3, V4, V5, and V6, respectively, which are electrical signals measured different locations of the heart.
As we mentioned above, ECG captures both spatio-temporal information. Initially, it provides tem- poral insights by monitoring the electrical activity of the heart continuously. This is achieved through the detection of voltage fluctuations within the cardiac muscle during each cardiac cycle, which are then represented as waveforms over time. Consequently, it displays multiple cycles of heart electri- cal activity spanning from one beat to the next. Moreover, ECG is not just single measurement but rather involves multiple leads. Leads are like different views of the heart's electrical activity from different angles. In a standard 12-lead ECG, there are 12 different leads, each offering a distinct cardiac electrical activity. The limb leads (I, II, III, aVR, aVL, and aVF) are placed the arms and legs, providing frontal plane views, and the precordial leads (V1 to V6) are placed on the chest, giving anterior poatiioo-potterooo(r horizontal plane) views.
spatial perspective
Although the standard 12-lead ECG remains significant, the utilization its reduced lead sets should be considered as well. The advancement of mobile devices, such as smartwatches that are capable of ECG measurements, has led to substantial increase in limb leads and single lead data. Obtaining reduced lead sets like limb leads or single lead is often preferred. Therefore, in this work, we demonstrate our ECG representation not only on the 12-lead setting but also on reduced lead settings.
METHOD
3.1 SELF-SUPERVISER PRE-TRAINING WITH TEMMPOLTL MASKED AUTO-ENCODER
In this section, we present ST-MEM, self-supervision framework for general representation learn- ing for ECG. This incorporates the spatio-temporal relationship. which leads to providing enhanced representation. An overview of our proposed method depicted Figure
Masked auto-encoder with spatio-temporal patchifying. We consider pre-text task to recon- struct randomly masked portion of the data. particular, we adopt an auto-encoder-basee re- construction task, called MAE (He et al., 2022). It consists of vision transformer (ViT) encoder Dosovitskiy et al. 2020) and a decoder with additional transformer blocks.
Considering ECG signal denoted as X E L with L leads and length T, each /th lead signal, represented as X is divided into non-overlapping patches defined as Patch = (Patch'..... Patch", } E R"xp, Here, is determined by T/p, and p represents the size of each patch. We independently split each lead signal into apatio-tempore patches, as depicted in Figure 2 (c), resulting in total of I. patches for single 12-lead ECG signal.
3

---

Published as a conference paper at ICLR 2024
The patches undergo linear projection to form patch embeddings of dimension D. They are added with a positional embedding, Pos E IR,, to create a embedding sequence as = = 1...... Z E R R.xxxD. This embedding sequence becomes the imput for the encoder. Since self-attention opera- tions are executed among temporal patches across different leads, the encoder explicitly learns the spatio-temporal relationship within the ECG data. In the pre-training phase, we randomly select in- dices of embeddings, which are denoted as masked index set M. After masking the corresponding embeddings, the encoder only receives unmasked embeddings, {Zi JigM, where the number of un- masked embeddings is determined as Ln(1 m) with masking proportion m. We set the masking ratio a value in [0, 1].
The encoded embeddings are fed to the decoder along with a learnable shared mask embedding, defined by Masked E RRD. This embedding is employed to reconstruct the patches of the M. The training objective is to minimize the discrepancy between the original raw signal val- ues of the masked patches, (Paach;Jiea) E (pm))xx and their corresponding reconstructions, Patch; - Patch;
(aatch:)eM.. The loss function is defined as LSsL Lam ΣiEM
Lead-wise shared decoder. Although spatio-temporal patchifying can have benefits in ECG repre- sentation learning, it can be detrimental without careful consideration, especially within the MAE framework. One drawback of spatio-temporal patchifying is that it simplifies the reconstruction task, allowing the decoder to access unmasked embeddings from different leads with identical temporal information. This ease of reconstruction can negatively impact the training of the encoder.
To address this issue, we employ a straightforward approach by limiting the decoder to operate on only one lead. We configure the decoder to process the embedding sequence of each lead indepen- dently, ensuring that it does not make explicit use of embeddings from other leads during recon- struction. This intentional design choice introduces complexity to the task, motivating the encoders to effectively learn spatio-temporal representations. Additionally, we promote training efficiency by having patch embeddings each lead share a single decoder.
Lead indicating modules. Merely increasing the difficulty of the reconstruction task may not be sufficient to enable our encoders to learn spatial representations effectively. Specifically, our encoder and decoder may face challenges in differentiating the lead from which the embedding originates. To address this, we introduce additional modules aimed at enhancing the ability to distinguish between leads effectively.
First, we add a spedifiecffic embedding. denoted as Lead E RD,, for every patch embedding. This lead embedding is function of the originating lead of the patch embedding, ensuring that all patch embeddings originating from the same lead have the same lead embedding. Next, we insert an additional shared embedding. denoted [SPP] E RDD, which is added before and after each patch embedding sequence. These modules support the model to distinguish between patch embeddings from different leads (i.c., leading the model learn the relationships between the different leads).
3.2 DOWNSTREAM FINE-TUNING
After pre-training through SSL, we utilize only the encoder for downstream tasks and discard the decoder. We augment the encoder with basic linear layer, which serves the classifier head. Subsequently, the encoder, along with the classifier head, is fine-tuned to optimize performance on downstream data across various scenarios.
ECG classification. For a multi-class arrhythmia diagnosis problem with K classes, our model takes the ECG input, encodes it with the encoder, and passes it through the classifier head to calculate logits. The model makes class predictions, ý = (.,,...,?i ỘK}, J. by applying the softmax function to the logits. These predictions are then compared with the actual one-hot class label y {yI.-.., 9K.
to minimize the loss function defined as LCE = k Σ (-y. log().)).
Fine-tuning in reduced lead setting. Our model demonstrates robust adaptability, even when there a discrepancy in the number of leads between datasets used for pre-training and fine-tuning. Firstly, the shape of the input patch fed into model remains unaffected by the number of leads, making our model agnostic to the number of leads in the input ECG. Additionally, our model learns distinctive characteristics of different leads through the lead-indicating module. As a result, our
4

---

Published as a conference paper at ICLR 2024
model is well-suited for reduced lead scenarios such as pre-training 12-lead dataset to learn general representations and subsequently fine-tuning on a dataset with fewer leads.
4 EXPERIMENTAL SETTINGS
This section provides a brief overview of the datasets utilized in our experiments, well as the base- lines against which our proposed method will be compared. More detailed information regarding the datasets, baselines, and implementation can be found in Appendix A
4.1 DATASETS
There are three 12-lead ECG datasets designated for pre-training, along with three other ECG datasets intended for downstream tasks. In the context of pre-training. we leverage the combined power of all three datasets. These datasets consist of Chapman (Zheng et al. (2020b). Ningbo (Zheng al.] (2200)), and CODE-15 (Ribeiro al. (02T). As for the downstream datasets, we used PTB- XI. (Wagner et al., 2020), CPSC2018 (Liu et al. (018). and PhysioNer2017 (Clifford et al., 2017) datasets, where PhysioNer2017 is a single-lead dataset (i.e., lead dataset) and others are 12-lead ECG datasets. For consistency, we resampled all ECGs, including pre-training and downstream datasets, 250Hz.
Pre-training datasets. Chapman dataset comprises 10,646 12-lead ECG recordings, each lasting 10 seconds with sample rate of 500 Hz. Similarly, the Ningbo dataset contains 34.905 12-lead ECG recordings, also 10 seconds in duration and sampled at 500Hz. On the other hand, CODE-15 encompasses 345.779 12-lead ECG recordings from 233,770 patients, with a sample rate of 400 Hz. Within the 345,779 ECGs, 143,328 are 10-second recordings, with the remainder being 7 seconds, and we and we exclusively utilized the 10-second ECGs. In alignment with the PhysioNet 2021 challenge (Reyna et al. 202I), we removed 399 ECGs from the Chapman dataset. We proceeded to merge these three datasets into a unified dataset for pre- tre-training. This pre- mee-trinnng dataset comprises total of 188,480 ECGs. Note that, during pre-trate pre-training, use all ECGs without focusing on specific labels.
Downstream datasets. PTB-XL comprises 21,837 12-lead ECG recordings collected from 18,885 patients, each lasting 10 seconds and sampled at a rate of 500 Hz. This dataset has five distinct labels, including cardiac arrhythmia and myocardial infaretion. In the case of CPSC2018, it contains 6,877 12-lead ECG recordings, ranging from 6 seconds to 60 seconds in duration, with a sample rate of 500 Hz. This dataset characterized by nine different labels associated with cardiac arrhythmia. Lastly, PhysioNer2017 consists of 8,528 ECGs recorded from lead I, with durations spanning from 9 60 seconds and sampled at a rate of 200 Hz. This dataset has four cardiac arrhythmia categories. ECGs shorter than 10 seconds are omitted, and for the remaining records, each is cropped into consistent length of disjointed 10 seconds. These cropped ECG segments are considered individual data points and evaluated independently.
4.2 BASELINES
All backbones are fixed as ViT-B (Dosovitskiy et al. 2020) with different patch projection layers corresponding to the patchifying way. We compared our pre-training method to networks initialized randomly (i.e., Supervised) and pre-trained models that learn representation by contrastive learning and generative learning methods. In the realm of contrastive learning, we evaluated both MoCo v3 (Chen et al. (202T) and Contrastive Multi-segment Coding (CMSC) from CLOCS (Kiyasseh) et a 2021). MoCo v3 encourages the similarity between representations of instances and their augmented counterparts, while CMSC divides an ECG into two temporal segments and encourages the similarity of representations for these segments.
Additionally, we examined generative learning methods, specifically Masked Time Autoencoder (MTAE) and Masked Lead Autoencoder (MLAE) from MaeFE (Zhang et al.) 2022). MTAE con- structs representations by reconstructing temporal patches as depicted in Figure (a). while MLAE does by reconstructing spatial patches, illustrated in Figure 2 (b). These models do not have lead- indicating modules (c.g., lead-wise wise decoder, SEP, and lead embeddings) and differ in their approach to patchifying input ECGs. Furthermore, to enhance robustness and adaptation reduced lead sets,
5

---

Published as a conference paper at ICLR 2024
Table 1: Linear evaluation and fine-tuning results of arrhythmia and myocardial infarction (MI)
classification tasks. The experiment is conducted based on 12-lead ECG data on unseen data (i.e., not used during the pre-training stage).
PTB-XI.
CPSC2018
Methods
Accuracy
FI
AUROC
Accuracy
FI
AUROC
Supervised
0.787 # 0.006
0.004 # 0.010
0.905 = 0.004
0.779 # 0.008
0.753 = 0.012
0.958 = 0.002
Linear Evaluation
MoCo v3
0.352 # 0.000
0.142 # 0.000
0.739 = 0.006
0.268 # 0.055
0.080 = 0.038
0.712 = 0.054
CMSC
0.681 0.032
0.441 0.058
0.797 0.038
0.361 0.005
0.238 = 0.022
0.724 0.013
MTAE
0.683 0.008
0.437 + 0.012
0.807 = 0.006
0.486 + 0.012
0.349 = 0.034
0.818 0.010
MIAE+RLM
0,687 0.006
0.444 # 0.009
0,806 = 0.005
0.480 # 0.010
0.342 0.022
0.824 = 0,006
MLAE
0.649 0.008
0.382 0.020
0.779 0.008
0.443 0.014
0.263 0.021
0.794 0.016
ST-MEM (Ours)
0.726 + 0.005
0,508 # 0.008
0.838 = 0.011
0.723 # 0.008
0.641 = 0.010
0,938 = 0.002
Fine-tuning
MoCo v3
0.799 + 0.004
0.644 # 0.010
0.913 = 0.002
0.852 # 0.002
0.838 = 0.002
0.967 # 0.003
CMSC
0.724 + 0.067
0.1115
0.877 0.003
0.736 0.006
0.717 0.006
0.938 0.006
MTAE
0.789 = 0.002
0.613 0.015
0
0.793 0.004
0.769 0.004
0,961 = 0.001
MTAE+RLM
0,793 0.002
0.615 + 0.007
0,911 = 0.004
0.782 + 0.002
0,756 = 0.003
0,960 : 0,002
MLAE
0.802 0.004
0.625 # 0.009
0.915
0.834 0.007
0.816 0.009
0.973 0.002
ST-MEM (Ours)
0,825 + 0.002
0.655 # 0.003
0.933 = 0.003
0.872 # 0.009
0.857 = 0.012
0.980 = 0.001
we introduce augmentation Random Lead Masking (RLM) MTAE+RLM.
(Oh et al. 2022 to MTAE, resulting in
EXPERIMENTS AND RESULTS
In this section, examine the results of our experiments, evaluating them both quantitatively and qualitatively to verify the effectiveness of ST-MEM. Additional experimental results are reported in Appendix B
5.1
EXPERIMENTAL RESULTS
As shown in Table 1 we evaluate the general ECG representation obtained from each SSL method by conducting both linear evaluation and fine-tuning experiments on unseen datasets (i.e., not used during pre- treerraiinng stage). PTB-XL is task of classifying both myocardial infarction (MI) and car- diac arrhythmia, while CPSC2018 consists of only cardiac arrhythmia [ Our proposed method, ST- MEM, shows outperforming performance against other baseline methods. In particular, we achieve a non-trivial margin with others on accuracy, F1, and AUROC scores in the linear evaluation. For instance, the Fl score is increased in the range of 0.064 to 0.366 and 0.292 to 0.561 both PTB- XL and CPSC2018 datasets. This demonstrates that ST-MEM learns general ECG representation compared others by explicitly considering both spatial and temporal information for ECG sig- nals. Moreover, we present the applicability of ST-MEM by adapting to each different heart disease task. Although the baseline methods pre-trained from large ECG datasets, their performance is similar to the supervised learning method, only used with downstream datasets, PTB-XI. and CPSC2018. The CMSC pre-training method is even lower than the supervised learning method when we fine-tune the encoder. However, ST-MEM shows consistent outperforming performance on task adaptation, which is fine-tuning on downstream datasets.
5.2 EFFECTIVENESS OF GENERAL ECG REPRESENTATION IN A LOW-RESOURCE SETTING
The diversity of heart diseases and few experienced cardiologists prevent obtaining abundant la- beled ECG data; thus, relying only on supervised learning methods is not a suitable solution for the low-resource problem. We conduct low-resource experiments to demonstrate that general ECG representation can mitigate the scarcity labeled data. As shown in Table we first randomly sam- pled 1% and 5% of data from each dataset. Then, fine-tune the model for each task. Although
CCardiac arrhythmia can he divided into different sub-classes (e.g., AV block (AVB) and premature atrial contraction (PAC)) in both PTB-XI. and CPSC2018 datasets.

---

Published as a conference paper at ICLR 2024
Table 2: Experiments of low-resource settings. 1% and 5% indicate the random sampling for training and validation data; however, the test data are the same for all results. Three different sampling was done, and the results were averaged. 12-lead ECG signals are used for each result, and the score represents the AUROC scores.
PTB-XI.
CPSC2018
Methods
1%
5%
100%
1%
5%
100%
Supervised
0.676 = 0.011
0.736
0.020
0.905 # 0.004
0 0.095
0.609
0.111
0.958
0.002
MoCo v3
0.797 = 0.006
0.826 = 0.015
0.913
0.791 0 0.045
0.903 0.019
0.967 = 0.003
CMSC
0.648 = 0.064
0.773 # 0.023
0.877 +: 0.003
0.625 4 0.013
0.732 # 0.038
0.938 + 0.006
MTAB
0.707 = 0.024
0.713 + 0.001
0,910 + 0.001
0.670 + 0.032
0.756 0.013
0.961 + 0.001
MTAE+RLM
0.730 = 0.030
0.730 # 0.003
0,911 + 0.004
0.708 + 0.020
0.726 = 0.011
0.960 = 0.002
MLAE
0.793 = 0.007
0.838 +: 0.018
0,915 0.001
0.860 + 0.013
0.922 = 0.007
0.973 + 0.002
ST-MEM (Ours)
0.815 0.012
0.878 0.011
0.933 = 0,003
0.897 = 0.025
0.952 0.004
0,980 + 0.001
Table 3: Robustness of any lead combinations. 6-lead represents limb leads, I, II. III. aVR, aVL, aVF, while I-lead is single lead, I. The bold represents the significant difference (p < 0.05) against other baseline methods. The score indicates the AUROC score.
PTB-XL
CPSC2018
PhysiaNei2011
Methods
12-lend
6-lend
I-lend
12-lead
6-lead
I-lead
1-lead
MTAE+RLM
0.911 = 0.004
0.888 z 0.002
0.795 # 0.003
0,960 a 0.002
0,931 = 0.017
0.909 a 0.006
0.857 a 0.005
MLAE
0.915 = 0.001
0.890 = 0.001
0.797 = 0.001
0.973 a 0.002
0,959 = 0.002
0.925 + 0.001
0.861 = 0.003
ST-MEM (Ours)
0.933 0.003
0.903 = 0.007
0.804 = 0.005
0,980 + 0.001
0.973 + 0.002
0.937 # 0.006
0.866 + 0.003
the performance of supervised learning method drops drastically compared to 100% of data, other SSL methods still show comparable performance, which shows the need for general ECG represen- tation learning. However, our proposed method, ST-MEM, surpasses other representation learning methods from all 1 % and 5%. Moreover, in CPSC2018, only with 5% of the data, we could achieve a similar performance against 100% of the data result.
5.3 PERFORMANCE IN REDUCED LEAD SETTINGS
12-lead ECG is standard measurement that provides whole spatial and temporal information re- garding the heart. However, measuring precordial leads (i.e., VI to V6) requires expertise, while limb leads (i.e., I, II, III, avR, aVL, and aVF) are easily accessible through smart devices such as smartwatch. Therefore, we need a generally applicable ECG representation that is robust not only for 12 leads but also for any lead combinations. Simple representation learning methods, such as MoCo v3, CMSC, and MTAE, do not consider the robustness of the lead combination; thus, the model requires 12 lead signals during a fine-tuning stage. On the other hand, the random lead mask- ing (RLM) method is one of the approaches to handle any lead combinations since the model can learn the robustness of leads by masking the random leads during the pre-training stage. Our pro- posed method, ST-MEM, also addresses the lead combination through spatio-temporal patchifying with explicit lead embeddings. Table compares ST-MEM with RLM and MLAE methods on 1- lead (I), 6-lead (I, II, III, aVR, aVL, and aVF). and 12-lead. The results demonstrate that ST-MEM robust to any lead combinations, even in a PhysioNe12017 dataset.
5.4 ANALYSIS OF GENERAL ECG REPRESENTATION LEARNED FROM ST-MEM
Incorporating spatial information in the ECG representation is significant for ECG signals. As shown in Figure [ the ECG signals are spatially separated by frontal and horizontal planes; the former indicates the limb leads, while the latter represents the precordial leads. To validate the spa- tial information for the representation learned from ST-MEM, we visualize the embedding space using t-SNE (Van der Maaten & Hinton, 2008) as depicted in Figure 4 Each circle indicates the representation of one ECG sample in a particular lead. We utilize the Gaussian mixture model (GMM) (Rasmussen. 1999) to cluster those representations learned from ST-MEM. Interestingly, the samples assigned to a blue cluster are mostly limb leads, and II, whereas the orange cluster is formed with precordial leads, V2 and V3. This phenomenon demonstrates that the representation learned from ST-MEM incorporates the spatial information of ECG signals. In addition, in Table quantitatively measure the inclusiveness of spatial relationships. We first define the samples,
7

---

Published as a conference paper at ICLR 2024
Chapman
Ningbo
CODE-15
Figure 4: A I-SNE plot of ECG signal representation learned from ST-MEM. Each circle represents the single ECG signal representation with different leads. The ellipse with blue and orange indicates the Gaussian (i.e., cluster) obtained from the Gaussian mixture model (GMM).
Table 4: Accuracy of representation clustering on two different groups: limb and precordial leads. Each model provides all representations of ECG signals included in Chapman through a pre-trained encoder. Representations are clustered using the Gaussian mixture model (GMM), After clustering, each score represents the accuracy of whether the clusters are accurately grouped into limb and precordial leads. 12-lead ECG signals are used in this experiment.
Chapman
ST-MEM (Ours)
MTAE
MoCo v3
Accuracy
0.895
0.607
0.506
assigned to Gaussian, as predicted limb or precordial lead labels. Then, we compute the accuracy of whether the lead sample is correctly clustered into frontal (i.e., limb Ieads) horizontal (i.e., precordial leads) plane. Since the baseline methods, MTAE and MoCo v3, could not consider spa- tial information, each representation of the ECG signal is randomly assigned to cluster that causes low accuracy. However, ST-MEM shows high accuracy since samples are grouped in the frontal or horizontal plane.
5.5 ABLATION STUDY
Table 5 shows the importance of the lead indication mod- ule for ST-MEM. We conduct the linear evaluation in both PTB-XI. and CPSC2018 datasets and compute the AU- ROC score. Applying only spatio-temporal patchifying without considering the lead type yields the lowest per- formance. However, when we include lead-wise shared encoder and each lead indication module (e.g.. SEP em- bedding, and lead embedding), the results of linear evalu- ation monotonically increase a CPSC2018 dataset. This demonstrates the effectiveness of spatial information while learning the general ECG representation.
Table 5: Effectiveness of lead-wise in- dication module. Each score indicates the AUROC score.
Method
PTB-XL
CPSC2018
Sparintemporal patchifying
0,805
0.861
& Leud-wise shared decoder
0.822
0.911
SEP embedking
0.820
0.929
& Lead embedding (Ours)
0.838
0.938
5.6 INTERPRETATION OF ST-MEM BY ANALYZING AETENIITNNS
By analyzing self-attention, we could observe that ST-MEM considers the ECG signal spatially and temporally. Figure an attention map for single lead III query patch (i.e., red dashed box). Since lead III is spatially located in frontal plane, the attention scores are generally high for limb leads. This phenomenon explains why the ECG representation
clustered in the frontal or horizontal plane. Moreover, from temporal perspective, the patches show high attention scores if the signal shape is similar to the query patch in that the ECG signal contains a periodic rhythm. Overall, we could observe that ST-MEM incorporates both spatial and temporal relationship of ECG signals. In addition, instead of patchifying temporally or spatially, the apatio-tempore patchifying can also benefit the cardiologist in diagnosing heart disease by analyzing the spatio-temporal self-attentions.
8

---

Published as a conference paper at ICLR 2024
6 RELATED WORKS
Our work is closely related to the recent work on self-supervised learning, such contrastive and generative learning, used widely in computer vision and natural language processing.
ECG on self-supervised learning. Recently, research employing deep learning for ECG analysis has extended wide range of tasks, including the diagnosis of cardiac diseases such as arrhyth-
mia classification (Ribeiro et al. 2020 Hu et al.] 2022 Jun et al. 2018: SStrodhhoff et al. 2020) emotion recognition (Sarkar & Etemad 2020), and patient identification (Oh et al. 2022 Li et al,
2020). Furthermore, self-supervised learning utilized to learn general representations, with some studies focusing on contrastive learning methods. In contrastive learning, some papers apply con- cepts such semantic preservation after data augmentation (Chen et al., 2020 He et al. to ECG data (Lai et al. 2023). Additionally, there a study on the utilization of temporal invan- ance and spatial invariance within a single ECG data to preserve semanties (Kiyasseh et al. 2021 Some research combines CNN and transformer architectures to learn local and global features from ECG data (Oh al. 2022). In terms of generative learning. MaeFE (Zhang al., 2022) applies MAE (He et al. (022), which patchifies temporally or spatially for analysis of ECG. Morcover, some studies (Sawano et al. 2022) were conducted by applying spatio-temporal patchifying. shown in Figure 2 (c). to acquire general ECG representation, but without lead-indicating modules, it is challenging learn the spatial relationship between leads.
2020
Challenging on reduced lead ECG. Obtaining a standard 12-lead ECG, however, can be excessive and often requires high-level clinical knowledge (Giannetta et al.] (222)) that may not be readily available. Therefore, recent advancements in ECG technologies have resulted in the creation of smaller and portable devices that can obtain the reduced lead set of 12- lead ECG. This advancement promotes the research ply a machine learning technique to a reduced lead ECG set- ting (Kiyasseh al., 2021; Hannun et al.) 2019 Urtnasan eet al. [022, Mathews et al. 2018). It is worth noting that the reduced-lead ECG dataset less abundant compared to 12-lead ECG, and there is research dedicated to addressing this limitation.
ap-
the context of self-supervised learning, RLM (Oh et al.] (022) was introduced to acquire robust repre- sentation to input ECG lead combinations. On the other hand, in Knowledge Distillation approaches. there is work that uti- lizes a model preitedi on 12-lead ECG as a teacher model and introduces student model that takes single lead in- put (Qin et al. 2023). This student model leverages the representation from the 12-lead ECG teacher model.
Figure 5: An illustration of an at- tention map. Attention scores from each encoder layer and head are averaged.
7 CONCLUSION
In this work, we propose ST-MEM, Temporl Masked Electrocardiogram Modeling, to learn the general ECG representation, generally applicable diverse ECG problems by incorporating the spatial and temporal relationship of ECG signal. Through extensive experiments, we first demon- strate that ECG representation learned from ST-MEM exhibit spatial relationships. Since limb leads are measured in the frontal plane, while precordial leads are in the horizontal plane, each representation from ST-MEM reflects that spatial information. Second, we observe the temporal relationship through an attention map. Third, we evaluate the general ECG representation in various experiments, such as reduced lead and low-resource settings. We believe that our work, providing general ECG representation encapsulating the spatio-temporal relationship, can benefit the recent healthcare industry. As part of our future work, we aim to explore the application of ST-MEM to general multivariate time-series datasets. Our experimental results on the human activity recogni- tion (Anguita et al.] 2013) task can be found Appendix C Moreover, we will explore the vari- ous transfer learning approaches (e.g., partially fine-tuning the pre-trained model (He et al. 2022)) to adapt to downstream tasks using general ECG representation instead of naively fine-tuning the model.

---

Published as a conference paper at ICLR 2024
8 REPRODUCIBILITY STATEMENT
Our code encompasses the implementation of ST-MEM and several other baselines written in Python. Furthermore, we furnish the pre-trained model parameters facilitate others in fine-tuning the model and achieving reproducible results. Moreover, comprehensive details on training hyper- parameters, schemes, and hardware specifications are provided. While we provide just one down- stream dataset, PTB-XL, already preprocessed and partitioned into train, validation, and test sets, we ensure access to additional datasets via provided links.
8.1 CODES
The codes will be available at "README.md" file and follow the instructions.
Please read the
8.2
HYPERPARAMETERS
All details in pre-training, fine-tuning, and linear evaluation can be found in Appendix A.1 It includes basic hyperparameters such as epochs, batch size, and optimizer.
8.3
DATASETS
In Secion 4.1 there all the information of datasets including the number of ECGs, the number of patients, sample rate, and its duration.
Moreover, all ECG data resampled to 250 Hz, and additional information can be found in Ap- pendix A.2 Moreover, one of the downstream datasets, PTB-XL, separated into the train, valid, and test datasets, can be downloaded from the link in our codes (README.md))
Although only preprocessed PTB-XL is given, all raw data can be downloaded from the below links:
PTB-XL, Chapman, Ningbo and CPSC2018 http::///bbyysllotttooooo/////
cha lenge 200iee-222/11.0.33
CODE-15: http://////.ooo/////////////22//
PhysioNer2017 httpss//hhpppaatttoooo//rrrreeer 0.0/
Ningbo and Chapman can be downloaded together in PhysioNet Challenge 2021 (45,15) 152 ECG recordings). For CODE-15, as we mentioned in Secion 4.1, ECGs with a duration of less than 10 seconds should be dropped out, and there are some zero padding at the start and end of the recordings, these zero-paddings should be stripped out.
REFERENCES
Davide Anguita, Alessandro Ghio, Luca Oneto, Xavier Parra, Jorge Luis Reyes-Ortiz, al. public domain dataset for human activity recognition using smartphones. In Esann, volume 3, pp. 3, 2013.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models few-shot learners. Proc. of the Advances in Neural Information Processing Systems (NeurIPS), 33:1877-1901, 2020.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In Proc. the IEEE conference on computer vision and paltern recognition (CVPR), pp. 9650-9660, 2021.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In Proc. of the International Conference on Machine Learning (ICMI., pp. 1597-1607. PMLR, 2020.
10

---

Published as a conference paper at ICLR 2024
X Chen, S Xie, and K He. An empirical study of training self self-supervised vision transformers. in 2021 icee. In Proc. of the IEEE international conference on computer vision (ICCV), pp. 9620- 9629, 2021.
Gari D Clifford, Chengyu Liu, Benjamin Moody, H Lehman Li-wei, Ikaro Silva, Qiao Li, AE John- son, and Roger G Mark. Af classification from short single lead eeg recording: The phys- ionet/computing in cardiology challenge 2017. In 2017 Computing in Cardiology (CinC), pp- 1-4. IEEE, 2017.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An im- age is worth 16x16 words: Transformers for image recognition at scale. In Proc. of the Interna- tional Conference on Learning Representations (ICLR), 2020.
Noemi Giannetta, Giuseppe Campagna, Flavio Di Muzio, Emanuele Di Simone, Sara Dionisi, and Marco Muzio. Accuracy and knowledge in 12-lead eeg placement among nursing students and nurses: A web-based italian study. Acta Bio Medica: Atenei Parmensis, 91(Suppl 12), 2020.
Bryan Gopal, Ryan Han, Gautham Raghupathi, Andrew Ng, Geoff Tison, and Pranav Rajpurkar. 3kg: Contrastive learning of 12-lead electrocardiograms using ahysiologically-insiree augmen- tations. In Machine Learning for Health, pp. 156-167. PMLR, 2021.
Baan-Bnttien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent-a new approach to supf-supe learning. Proc. of the Advances in Neural Information Processing Systems (NeurIPS), 33:21271-21288 2020.
Changho Han, Youngjae Song, Moog-Seok Lim, Yunwon Tae, Jong-Hwan Hwan Jang, Byeong Tak Lee, Yeha Lee, Woong Bae, and Dukyong Yoon. Automated detection of acute myocardial infarction using asynchronous electrocardiogram
of implementing artificial intelligence with multichannel electrocardiographs obtained from smartwatches: retrospective study. Journal of Medical Internet Research, 23(9):331129, 2021.
Awni Y Hannun, Pranav Rajpurkar, Masoumeh Haghpanahi, Geoffrey H Tison, Codie Bourn, Mintu P Turakhia, and Andrew Y Ng. Cardiologist-lvvel arrhythmia detection and classifica- tion in ambulatory electrocardiograms using a deep neural network. Nature Medicine, 25(1):
65 65-69,20 2019.
Kaiming He, Haoqi Fan, Yuxin WI, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proc. of the IEEE conference on computer vision and pattern recognition (CVPR), pp. 9729-9738, 2020.
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In Proc. of the IEEE conference on computer vision and pattern recognition (CVPR), pp. 16000-16009, 2022.
Rui Hu, Jie Chen, and Li Zhou. A seaefformer-basee deep neural network for arrhythmia detection using continuous eeg signals. Computers in Biology and Medicine, 144:105325, 2022.
Rui Hu, Jie Chen, and Li Zhou. Spatiotemporal self-supervised representation learning from multi- lead ecg signals. Biomedical Signal Processing and Control, 84: 2023.
Longlong Jing and Yingli Tian. Self-supervised visual feature learning with deep neural networks: A survey. IEEE Transactions Pattern Analysis and Machine Intelligence (TPAMI). 43(11): 4037-4058, 2020.
Tae Joon Jun, Hoang Minh Nguyen, Daeyoun Kang. Dohyeun Kim, Daeyoung Kim, and Young- Hak Kim. Ecg arrhythmia classification using a 2-d convolutional neural network. arXiv preprint arXiv: /804.06812. 12, 2018.
Jacob Devlin Ming- Wei Chang Kenton and Lee Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proc. of The Annual Conference the North American Chapter of the Association for Computational Linguistics (NAACL), pp. 4171- 4186, 2019.
11

---

Published as a conference paper at ICLR 2024
Dani Kiyassch, Tingting Zhu, and David A Clifton. Clocs: Contrastive learning of cardiac signals across space, time, and patients. In Proc. of the International Conference on Machine Learning (ICML). pp. 5606-5615, PMLR, 2021.
Jiewei Lai, Huixin Tan, Jinliang Wang, Lei Ji, Jun Guo. Baoshi Han, Yajun Shi, Qianjin Feng, and
Wei Yang. Practical intelligent diagnostic algorithm for wearable 12-lead eeg via self- self--uupervied learning on large-scale dataset. Nature Communications, 14(1):3741, 2023.
Xiang Lan, Dianwen Ng. Shenda Hong, and Mengling Feng. Intra-inter subject self-supervised learning for multivariate cardiac signals. In Proc. the the AAAI Conference on Artificial Intelligence (AAAT), volume 36, pp. 4532 4540, 2022.
Xiang Lan, Hanshu Yan, Shenda Hong, and Mengling Feng. Towards better time series contrastive learning: A dynamic bad pair mining approach. arXiv preprint arXiv:2302.03357, 2023.
Duc Le, Sang Truong, Patel Brijesh, Donald Adjeroh, and Ngan Le. scl-st: Supervised contrastive learning with semantic transformations for multiple lead eeg arrhythmia classification. IEEE Journal of Biomedical and Health Informatics (JBHI), 2023.
Ke Li, Abdelmotagaly Elgalad, Cristiano Cardoso, and Emerson C Perin. Using the apple watch to record multiple-lead electrocardiograms in detecting myocardial infarction: Where are we now? Texas Heart Institute Journal, 49(4)22 2022.
Yazhao Li, Yanwei Pang. Kongqiao Wang, and Xuelong Li. Toward improving ecg biometric iden-
tification using cascaded convolutional neural networks. Neurocomputing. 391:83-95, 2020.
Feifci Liu, Chengyu Liu, Lina Zhao, Xiangyu Zhang, Xiaoling Wu, Xiaoyan Xu, Yulin Liu, Caiyun Ma, Shoushui Wei, Zhiqiang He, al. An open access database for evaluating the algorithms of electrocardiogram rhythm and morphology abnormality detection. Journal of Medical Imaging and Health Informatics, 8(7):1368 133-13333,7, 2018.
Sherin M Mathews, Chandra Kambhamettu, and Kenneth E Barner. A novel application of deep learning for single-lead ecg classification. Computers in Biology and Medicine, 99:53-62 62, 2018.
Temesgen Mehari and Nils Strodthoff. Self-supervised representation learning from 12-lead ecg data. Computers in Biology and Medicine, 141:105114 14, 2022.
Naoki Nonaka and Jun Seita. Data augmentation for electrocardiogram classification with deep neural network. arXiv preprint MxXiv:2009.04399 2020.
Jungwoo Oh, Hyunseung Chung. Joon-myoung Kwon, Dong-gyue gyun Hong, and Edward Choi. Lead- agnostic self-supervised learning for local and global representations of electrocardiogram. In Proc. of Conference on Health, Inference, and Learning (CHIL), pp. 338-353. PMLR, 2022.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predic- tive coding, arXiv preprint
2018.
Namuk Park, Wonjae Kim, Byeongho Heo, Tackyung Kim, and Sangdoo Yun. What do self- supervised vision transformers learn? In Proc. of the International Conference Learning Representations (ICLR), 2022.
Yuzhen Qin, Li Sun, Hui Chen, Wei-qiang Zhang, Wenming Yang, Jintao Fei, and Guijin Wang. Mvkt-ecg: Efficient single-lead ecg classification on multi-label arrhythmia by multi-view knowl- edge transferring. arXiv preprint arXiv:2301.12117 2023.
Carl Rasmussen. The infinite gaussian mixture model. Proc. of the Advances in Neural Information Processing Systems (NeurIPS), 12, 1999.
Matthew A Reyna, Nadi Sadr, Erick Perez Alday, Annie Gu, Amit Shah, Chad Robichaux, Ali Bahrami Rad, Andoni Elola, Salman Seyedi, Sardar Ansari, et al. Will two do? varying dimensions in electrocardiography: the physionet/computing in cardiology challenge 2021. In 2021 Computing in Cardiology (CinC), volume 48, pp. 1-4. IEEE, 2021.
12

---

Published as a conference paper at ICLR 2024
AH Ribeiro, GM Paixao, EM Lima, et al. Code-15%: A large scale annotated dataset of 12-lead ecgs. Zenodo, Jun, 9, 2021.
António H Ribeiro, Manoel Horta Ribeiro, Gabriela MM Paixão, Derick M Oliveira, Paulo R Gomes, Jéssica A Canazart, Milton PS Ferreira, Carl R Andersson, Peter W Macfarlane, Wagner Meira Jr, et al. Automatic diagnosis of the 12-lead ecg using a deep neural network. Nature Communicationss 11(1):1760, 2020.
Pritam Sarkar and Ali Etemad. Self-supervised eeg representation learning for emotion recognition. IEEE Transactions Affective Computing (TAC), 13(3):1541-1554 2020.
Shinnosuke Sawano, Satoshi Kodera, Hirotoshi Takeuchi, Issei Sukeda, Susumu Katsushika, and Is- sei Komuro. Masked autoencoder-based self-supervised learning for electrocardiograms to detect left ventricular systolic dysfunction. In NeurlPS 2022 Workshop on Learning from Time Series for Health, 2022.
Konstantinos C Siontis, Peter A Noseworthy, Zachi I Attia, and Paul A Friedman. Artificial eeeeligence-enhancer electrocardoorraphy in cardiovascular disease management. Nature Re- views Cardiology, 18(7):465 478, 2021.
Sahar Soltanich, Ali Etemad, and Javad Hashemi. Analysis of augmentations for contrastive ecg representation learning. In Proc. of International Joint Conference on Neural Networks (IJCNN), pp. 1-10. IEEE, 2022.
Nils Strodthoff, Patrick Wagner, Tobias Schaeffter, and Wojeiech Samek. Deep learning for eeg analysis: Benchmarks and insights from ptb-xl. IEEE Journal of Biomedical and Health Infor- matics (JBHI), 25(5):1519-1528 2020.
Erdenebayar Urtnasan, Jong-Uk Park, Jung-Hun Lee, Sang-Baek Koh, and Kyoung-Joung Lee. Deep learning for automatic detection of periodic limb movement disorder based on electrocar- diogram signals. Diagnostics,, 12(9):2149, 2022.
Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of Machine Learning Research (JMLR), 9(11), 2008
Patrick Wagner, Nils Strodthoff, Ralf-Dieter Bousseljot, Dieter Kreiseler, Fatima Lunze, Wojeiech Samek, and Tobias Schaeffter. Ptb-xl, large publicly available electrocardiography dataset. Scientific Data, 7(1):154, 2020.
Crystal T Wei, Ming-En Hsieh, Chien-Liang Liu, and Vincent Tseng. Contrastive heartbeats: Contrastive learning for self-supervised eeg representation and phenotyping. In Proc. of Interna- tional Conference on Acoustics, Speech, and Signal Processing (ICASSP), pp. 1126 1126-1130. IEEE, 2022.
Zhenda Xie, Zheng Zhang, Yue Cao. Yutong Lin, Jianmin Bao, Zhuliang Yao, Qi Dai, and Han Hu. Simmim: A simple framework for masked image modeling. In Proc. of the IEEE conference on computer vision and pattern recognition (CVPR), pp. 9653-9663. 2022.
Huaicheng Zhang, Wenhan Liu, Jiguang Shi, Sheng Chang, Hao Wang, Jin He, and Qijun Huang. Maefe: Masked autoencoders family of electrocardiogram for self-supervised pretraining and transfer learning. IEEE Transactions on Instrumentation and Measurement (TIM), 72:1-15, 2022.
Jianwei Zheng, Huimin Chu, Daniele Struppa, Jianming Zhang, Sir Magdi Yacoub, Hesham EI- Askary, Anthony Chang, Louis Ehwerhemuepha, Islam Abudayyeh, Alexander Barrett, et al. Optimal multi-stage arrhythmia classification approach. Scientific Reports, 10(1):2898, 2020a.
Jianwei Zheng, Jianming Zhang, Sidy Danioko, Hai Yao, Hangyuan Guo, and Cyril Rakovski. A 12-lead electrocardiogram database for arrhythmia research covering more than 10,000 patients. Scientific Data, 7(1):48, 2020b.
13

---

Published as a conference paper at ICLR 2024
A DETAILS IN EXPERIMENTAL SETTINGS
In this section, we provide in-depth explanations of our experimental setup, including our ap- proach to dataset processing, model evaluation on downstream tasks, and implementation of baseline methodologies.
A.1 IMPLEMENTATION DETAILS
All backbones fixed ViT-B with patch size 75 and different patch projection layers corre- sponding to the patchifying way as shown in Figure 2 temporal patchifying is adopted in MoCo v3, CMSC, and MTAE pre-training; spatial patchifying is used in MLAE pre-training: ST-MEM and its variants for ablation study use spatio-temporal patchifying. We use fixed sinusoid temporal positional embedding. while other special embeddings (e.g. mask, lead, and SEP embedding) are learned during training.
For contrastive pre-training, InfoNCE (Oord al. 2018 loss is used for MoCo v3 and CMSC pre-training. In CMSC pre-training, we divide ECG signals with length T into two temporal non- overlapped segments with length T/2 to make positive pairs. In MTAE, MLAE, and ST-MEM pre-training, we mask 75% randomly selected patches and reconstruct them. Mean squared error loss used for these generative pre-training. We fix the decoder design for all masked auto-encoder variants four transformer blocks, four heads, and 256 width. For RLM, randomly mask each lead with probability 0.5, Further details of hyperparameters used each pre-training is shown in Table
Table 6: Hyperparameter settings.
Pre-training
Fine-tuning
Linear evaluation
Backbone
ViT-B
ViT-B
ViT-B
Learning rate
0.0012
0.001
0.001
Batch size
2048
1024
32
Epochs
800
100
100
Optimizer
AdamW
A
Adam
Learing rate scheduler
Cosine anealing
Cosine anealing
Cosine anealing
Warump steps
40
5
5
For environment details, experiments examined with Ubuntu 20.04.6, AMD EPYC 7502 32- Core Processor, and NVIDIA GeForce RTX 3080 Ti. The version of the libraries we used in all experiments are 3.9.13 for Python and 1.11.0 for PyTorch.
A.2 DATA PREPROCESSING
Describing target labels of each downstream dataset. As mentioned earlier, we have three down- stream datasets: PTB-XL, CPSC2018, and PhysioNer2017. In Table there are 44 diagnostic labels for PTB-XL, each with its corresponding description. These 44 labels are merged into 23 subclass labels, and these 23 subclass labels are further merged into 5 superclass labels (Wagner al. 2020). The 5 superclass labels are Myocardial Infarction (MI), Conduction Disturbance (CD), ST/T- ST/T-Change (STTC). Hypertrophy (HYP). and Normal ECG (NORM). These 5 labels are our target labels in PTB-XI.. Table 8 provides explanations for the 9 arrhythmia target labels in CPSC2018 (Liu et al.
2018), while Table contains descriptions for the target labels the PhysioNer2017 dataset (Clif- ford et al. 2017).
ECG signal processing We have ECG datasets, each with different characteristics. For instance, Chapman and CODE-15 have varying sampling frequencies (500Hz and 400Hz) and units (micro- volts and millivolts). These differences data resolution and scale could potentially confuse the model and diminish prediction performance. To ensure consistency among datasets and improve ECG signal quality, we implemented several signal processing procedures. Initially. we resampled all ECGs to uniform sampling frequency of 250Hz. Next, applied bandpass digital filter
14

---

Published as a conference paper at ICLR 2024
with range of 0.67-40Hz to eliminate baseline wandering and high-freegg feehhffreueeey noise. Finally, normalized the ECG signals using Z-normalization to ensure uniform scales. The signal processing was conducted for all baselines regardless of whether it was during the pre-training or fine-tuning
stage.
Cropping ECGs into disjoint 10-second segments. Chapman, Ningbo, CODE-15 and PTB-XL consist 10-second ECG data, except for CPSC2018 and PhysioNer2017. To standardize all ECG data to 10 seconds, data with duration of less than 10 seconds are discarded, and only data ex- ceeding 10 seconds are utilized. For those ECGs more than 10 seconds, they are cropped into non-overlapping 10-second segments. Each crop is evaluated individually and tagged with an origin label for later use in loss computation (Zhang et al. 2022; Gopal et al. [021]] Lan et al., (022).
Removing ECGs which have more than one label for multi-class classification setting. A single ECG may exhibit multiple heart diseases simultaneously. i.e., one ECG could show both MI and CD concurrently. our ECG datasets, PTB-XL. and CPSC2018, some ECGs have more than one label. In multi-class classification setting, each ECG should have only one label. Consequently, we excluded ECGs with more than one concurrent label (Zhang al. 2022).
Dividing downstream datasets into train, validation and test set. Finally, regarding the down- stream datasets, they are divided into training, validation, and test sets, following 70- 70-10-20 config- uration. Table provides the preprocessing steps for PTB-XL, along with information about the uti- lized train, validation, and test sets. Likewise, Table IT presents information regarding CPSC2018, while Table 12 outlines details concerning PhysioNer2017.
A.3 EVALUATION ON DOWNSTREAM TASKS
The downstream task involves classifying arrhythmias and myocardial infarction. Due to imbalances in the labels within these downstream datasets, we assess model performance using metrics beyond accuracy, including F1 score and AUROC (Area Under the Receiver Operating Characteristic curve) for each experiment. Furthermore, since there are more than two classes involved, we compute the average of one-vs-rest AUROC and the macro Fl-score to provide comprehensive evaluation of the model's 's performance across multiple classes. As we mentioned before, all cropped ECG segments evaluated individually, neither averaging nor voting.
15

---

Published as a conference paper at ICLR 2024
Table 7: Description of each diagnostic label PTB-XL.
Label
Description
Subclass
Superclass
LAFB
left anterior fascicular block incomplete right bundle branch block
LAFB/LPEB
CD
IRBBB
IRBBB
CD
AVB
first degree AV block
AVB
CD
IVCD
non-specific intraventricular conduction disturbance (block)
IVCD
CD
CRBBB
complete right bundle brunch block
CRBBB
CD
CLBBB
complete left bundle branch block
CLBBB
CD
LPFB
left posterior fascicular block
LAFB/LPFB
CD
WPW
White syndrome
WPW
CD
ILBBB
incomplete left bundle branch block
ILBBB
CD
3VVB
third degree AV block
AVB
CD
2AVB
sccond degree AV block left ventricular hypertrophy left atrial overload/enlargement
AVB
CD
LVH
LVH
HYP
LAO/LAE
LAOILAE
HYP
RVH
right ventricular hypertrophy
RVH
HYP
RAO/RAE
right atrial overload/enlargement
RAO/RAE
HYP
SEHYP
septal hypertrophy
SEHYP
HYP
IMI
inferior myocardial infarction
IMI
MI
ASMI
anteroseptal myocardial infarction
AMT
MI
ILMI
inferolateral myocardial infarction
IMI
MI
AMI
anterior myocardial infarction
AMT
ALMI
anterolateral myocardial infarction
AMT
MI
INJAS
suhendocardial injury in anteroseptal leads
AMT
MI
LMI
lateral myocardial infarction
LMT
MI
INJAL
subendocardial injury in anterolateral leads
AMT
IPLMI
inferoposterolater myocardial infarction
IMI
MI
IPMI
inferoposterior myocardial infarction
IMI
MI
INJIN
subendocardial injury in inferior leads posterior myocardial infarction
IMI
MI
PMI
MI
INJLA
subendocardial injury in lateral Icads
AMI
MI
INJIL
subendocardial injury in inferolateral leads
IMI
MI
NORM
normal ECG
NORM
NORM
NDT
non-diagnostic abnormalities
STTC
STTC
NST-
non-specific ST changes
NST.
STTC
DIG
digitalis-effect
STTC
STTC
LNGQT
long QT-interval
STTC
STTC
ISC_
non- cete--iccifcc ischemic
ISC.
STTC
ISCAL
ischemic in anterolateral leads
ISCA
STTC
ISCIN
ischemic in inferior leads
ISCI
STTC
ISCIL
ischemic inferolateral leads ischemic in anteroseptal leads ischemic in lateral leads
ISCI
STTC
ISCAS
ISCA
STTC
ISCLA
ISCA
STTC
ANEUR
ST-T changes compatible with ventricular aneurysm electrolytic disturbance or drug (former EDIS)
STTC
STTC
EL
STTC
STTC
ISCAN
ischemic in anterior leads
ISCA
STTC
Table 8: Description of each label of CPSC2018.
Label
Description
NORMAL
normal ECG
AF
atrial fibrillation
LAVB
first-degree atrioventricular block
LBBB
left bundle branch block
RBBB
right bundle branch block
PAC
premature atrial contraction
PVC
premature ventricular contraction
STD
ST-segment depression
STE
ST-segment elevated
16

---

Published as a conference paper at ICLR 2024
Table 9: Description of each label of PhysiaNer2017.
Label
Description
NORMAL
normal sinus rhythm
AF
atrial fibrillation
OTHER RHYTHM
alternative rhythm
NOISY
too noisy to be classified
Table 10: The number of instances (ECGs) of downstream dataset PTB-XL.
ECGs
NORMAL
MI
STTC
CD
HYP
Original
21837
9528
5486
5250
4907
2655
After removing ECGs
16272
9083
2538
2406
1709
536
Train
11390
6379
1752
1706
1177
376
Valid
3255
1797
507
460
373
118
Test
1627
907
279
240
159
42
Table 11: The number of instances (ECGs) of downstream dataset CPSC2018.
ECGs
NORMAL
AF
LAVB
LBBB
RBBB
PAC
PVC
STD
STE
Original
6877
918
1098
704
207
1695
574
653
826
202
After cropping ECGs
9364
1201
1593
900
300
2357
1014
1256
1102
332
After removing ECGs
8682
1201
1266
840
225
1911
892
1104
971
272
Train
6077
849
865
585
161
1336
636
779
673
193
Valid
868
113
131
84
18
212
85
113
86
26
Test
1737
239
270
171
46
363
171
212
212
53
Table 12: The number of instances (ECGs) of downstream dataset PhysioNei2017.
# ECGs
Normal
AF
OTHER RHYTHM NOISY
Original
8528
5154
771
2557
46
After cropping ECGs
26940
15765
2317
8237
621
Train
18771
10946
1608
5778
439
Valid
2710
1537
217
875
81
Test
5459
3282
492
1584
101
17

---

Published as a conference paper at ICLR 2024
B ADDITIONAL EXPERIMENTS AND RESULTS
B.I COMPARISON OF EXISTING ECG PRE-TRAINED MODELS WITH ST-MEM
While we reproduced pre-trained baselines for comparison with ST-MEM, it is important to note that the pre-trained models may exhibit slight variations due to heterogeneous experimental settings, in- cluding diverse datasets, backbone encoder models, and hyperparameters. To further explore this, conduct additional experiments utilizzigg another pre-trained model, Contrastive Predictive Cod- ing (CPC) (Mehari & Strodthoff] 2022). which provides original pre-trained weights. As shown in Table 13 proposed approach, ST-MEM, demonstrates comparable performance CPC. More- over, in low-resource settings, certain baseline models like MIAAF and MoCo v3 exhibit similar performance, emphasizing the comparability of our reproduced pre-trained models as baselines. Additionally, it is worth mentioning that the CPC model employed PTB-XL and CPSC2018 as pre- trained datasets, whereas ST-MEM and other baselines treated these datasets as unseen datasets to validate the genuine performance of general ECG representation learning. Overall, we believe ST- MEM can provide general ECG representation to effectively solve diverse challenging tasks, e.g., diverse heart disease classification tasks, reduced lead settings, and low-resource settings.
Table 13: Experiments low-resource settings. CPC (Mehari & Strodthoff] 2022) indicates the original implementation of pre-trained model that utilizes pre-training datasets including PTB-XL and CPSC2018. 250 Hz and 100 Hz represent the sampling rate for ECG preprocessing during the fine-tuning stage. Note that CPC was pre-trained in sample rate of 100 Hz, yet default sample rate experimental setting is 250 Hz. Furthermore, 1% and 5% indicate random sampling for training and validation data, while the test data remain constant for all results. Three different samplings were conducted, and the results were averaged. 12-lead ECG signals are used for each result, and
the scores represent AUROC scores.
PTB-XL
CPSC2018
Methods
1%
5%
100%
1%
5%
100%
Supervised
0.676 + 0.011
0.736
0.020
0.905
0.004
0.600 1 0.095
0.609 0.111
0.958 = 0.002
MoCo v3
0.797 + 0.006
0,826 = 0.015
0.913 + 0.002
0.791 +: 0.045
0.903 1 0.019
0,967 = 0.003
CMSC
0.648 + 0.064
0.773 = 0.023
0.877 1 0.003
0.625 + 0.013
0.732 :: 0.038
0,938 = 0.006
MTAE
0,707 0.024
0.713
0.001
0.670 0.032
0.756 0.013
0.001
MTAE-RIM
0.730 + 0.030
0.730
0.003
0.911
0.004
0.708 + 0.020
0.726
0.011
0.960 + 0.002
MLAE
0.793 + 0.007
0.838
0.018
0.915 + 0.001
0,860 + 0.013
0.922 + 0.007
0,973 + 0.002
(250 Hz) CPCT
0.740 0.057
0.838
0.933
0.001
0.754 0.015
0.898
0.026
0.002
(100 Hz) CPCt
0.773 + 0.014
0.842
0.043
0.934 0.002
0.762 + 0.058
0.917
0.016
0.973 + 0.003
ST-MEM (Ours)
0.815 = 0.012
0,878
0.011
0.933 0.003
0,897 1 0.025
0.952 0.004
0.980 = 0.001
B.2 SELECTING AUGMENTATION FOR PRE-TRAINING MoCo v3
In MoCo v3 pre-training, positive pairs are made by employing eight straightforward sime-series augmentations (Nonaka & Seita 2020):
1. Erase: Randomly setting the values of chosen lead to 0.
2. Flip: Randomly inverting the signal vertically.
3. Drop: Randomly zeroing out signal values.
4. Cutout: Selecting random interval and setting its values to 0.
5. Shift: Randomly shifting the signal temporally.
6. Sine: Adding a sine wave to the entire signal.
7. Partial sine: Adding a sine wave into randomly selected interval.
8. Partial white noise: Introducing white noise into randomly chosen interval.
However, certain augmentations, such flip, shift, sine, and partial sine, may have negative impact on the semantics of the ECG. Flipping an ECG can result in misalignment with standard lead systems; shifting lead to a loss of temporal continuity; adding sine wave partial sine wave
18

---

Published as a conference paper at ICLR 2024
Table 14: Linear evaluation and fine-tuning performance of MoCo v3 PTB-XI. and CPSC2018.
PTB-XL.
CPSC2018
Methods
Accuracy
FI
AUROC
Accuracy
F1
AUROC
Linear Evaluation
MoCo v3, 4 augmentations 0.552
0.142
0.709
0.209
0.038
0.644
MoCo v3, 8 augmentations
0.552
0.142
0.739
0.268
0.08
0.712
Fine-tuning
MoCo v3, 4 augmentations 0.798
0.636
0.915
0.833
0.816
0.967
MoCo v3, 8 augmentations
0.799
0.644
0.910
0.852
0.838
0.967
to ECG may introduce distortion, altering the original morphology of the signal. Consequently, these changes can lead to misinterpretation of the ECG, potentially affecting the accuracy of diagnoses and analyses. In terms of contrastive learning, the distortion of semantic information makes it inappropriate to define positive or negative pairs.
Therefore, pre-train MoCo v3 using the remaining four augmentations (erase, drop, cutout, and partial noise) and compared the results with experiments using the original eight augmentations. After pre-training the same 12-lead dataset as outlined in the paper, fine-tune the model on 12-lead ECGs from PTB-XL and CPSC2018, with the results presented in Table [4] Using all eight augmentations performed better than using only four augmentations for all settings except for fine-tuning the model on PTB-XI. critical point emphasized here the challenge augmentations when working with ECG signals. This complexity highlights our preference for generative learning (masking and reconstructing) over contrastive learning which needs to select and use proper augmentations.
selecting
B.3 EXPERIMENTS OF DIFFERENT PROBLEM SETTINGS FOR DOWNSTREAM DATASET PTB-XI.
We expand problem to detect broader range of heart diseases through multi-label classification for other labels PTB-XL. PTB-XL consists and rhythm. As mentioned earlier, the diagnostic labels consist of 44 labels, which are merged into 23 subclasses, with their explanations provided Table 7 Additionally, there are 19 form labels and 12 rhythm labels, described in Table 15 and Table 16 respectively.
of consisss total of 71 labels, categorized into diagnostic, form,
We conducted multi-label classification for these three different settings, following a similar process in the preprocessing steps described in Appendix A excluding the step of dropping ECGs that have more than one label. We compared our approach, ST-MEM, with baselines such as supervised learning and MTAE. The superiority of our ST-MEM is highlighted Table 17
19

---

Published as a conference paper at ICLR 2024
Table 15: Description of each form label of PTB-XL.
Label
Description
NDT
non-diagnostic T abnormalities
NST.
non-specific ST changes
DIG
digitalis-effect
LNGQT
long QT-interval
ABQRS
abnormal QRS
PVC
ventricular premature complex
STD_
non-specific ST depression
VCLVH
voltage criteria (QRS) for left ventricular hypertrophy
QWAVE
Q waves present
LOWT
low amplitude T-waves non-specific T-wave changes
NT
PAC
atrial premature complex
LPR
prolonged PR interval
INVT
inverted T-waves
LVOLT
low QRS voltages in the frontal and horizontal leads
HVOLT
high QRS voltage
TAB
T-wave abnormality
STE.
non-specific ST elevation
PRC(S)
premature complex(es)
Table 16: Description of each rhythm label of PTB-XL.
Label
Description
SR
sinus rhythm
AFIB
atrial fibrillation
STACH
sinus tachycardia
SARRH
sinus arrhythmia
SBRAD
sinus bradycardia
PACE
normal functioning artificial pacemaker
SVARR
supraventricular arrhythmia
BIGU
bigeminal pattern (unknown origin, SV Ventricular) atrial flutter
AFLT
SVTAC
supraventricular tachycardia
PSVT
paroxysmal supraventricular tachycardia
TRIGU
trigeminal pattern (unknown origin, SV or Ventricular)
Table 17: Fine-tuniie tine-tuning average AUROC results for different settings of PTB-XL.
Categories
Methods
Subclass
Form
Rhythm
Supervised MТAE
0.914 = 0.002
0.829 + 0.021
0.934 + 0.003
0.911 0.001
0.793 + 0.014
0.920 + 0.005
ST-MEM (Ours)
0.929 = 0.001
0.895 + 0.008
0.966 + 0.004
20

---

Published as a conference paper at ICLR 2024
B.4 ADDITIONAL SINGLE LEAD EXPERIMENTS ON VARIOUS LEAD TYPES
Mobile devices such as smartwatches usually provide lead I ECGs. However, several studies (Li 2022, Han et al. 202T) show the possibility of obtaining other lead types single lead FCGGs Therefore, as shown in Table [8] we demonstrate the additional single lead experiments. Interestingly. ST-MEM still outperforms other baselines, MTAE+RLM and MLAE. Moreover, in lead II and a'L, our proposed approach, ST-MEM, surpasses the baselines with non-trivial margin. This shows that ST-MEM is not only applicable to 12-lead standard ECGs but also to single lead ECGs.
Table 18: Results of single lead on PTB-XL. Each value indicates the AUROC score.
PTB-XL
Method
Lead 1
Lead I1
Lead III
Lead avR
Lead aVL
Lead aVF
MTAE+RLM
0.795 + 0.003
0.833 = 0.002
0.742 + 0.005
0.839 0.002
0.753 0.006
0.801 0.003
MLAE
0.797 + 0.001
0.826 = 0.003
0.743 # 0.006
0.830 0.003
0.753 0.004
0.793 0.002
ST-MEM (Ours)
0.804 = 0.005
0.856 = 0.002
0.788 = 0.019
0.840 = 0.003
0.805 0.003
0.819 = 0.022
B.5 RESULTS OF SINGLE LEAD ECGS A LOW-RESOURCE SETTING
While smart devices readily supply unlabeled single lead ECGs, we believe acquiring small num- ber of labeled ECGs from cardiologists is feasible endeavor. As demonstrated in Table 19 we show the results of single lead ECGs in low-resource settings. Leveraging the capabilities of general ECG representation learning. ST-MEM consistently surpasses the performance of the supervised baseline model. Notably, ST-MEM achieves an AUROC score of 0.786 using only 5% of lead II ECGs.
Table 19: Experiments of low-resource settings using single lead ECGs. 1% and 5% indicate the random sampling for training and validation data; however, the data are the same for all results. Three different sampling was done, and the results were averaged. The score represents the AUROC scores.
PTB-XL
Method
Lead I
Lead II
Lead III
Lead aVR
Lead aVL
Lead aVF
1%
Supervised
0.638 # 0.035
0.657 = 0.025
0.538 = 0.05
0.668 # 0.021
0,562 # 0.015
0.603 # 0.03
ST-MEM (Ours)
0.673 0.012
0.752 0.03
0.640 0.008
0.681
0.008
0.667 0.006
0.674 0.01
5%
Supervised
0.655 = 0.018
0.682 = 0.006
0.577 = 0.041
0.654
0.053
0.593 0.003
0.652 0.012
ST-MEM (Ours)
0.694 = 0.017
0.786 = 0.024
0.700 + 0.016
0.691 0.012
0.718 0.026
0.723 0.024
B.6 ADDITIONAL ANALYSIS OF ECG REPRESENTATION LEARNED FROM ST-MEM
In order to validate the effectiveness of the lead indication module (e.g.. lead- wise shared decoder, SEP embedding, and lead embedding), we plot t-SNE for sampled 12-lead ECG signals across regular and irregular rhythms which are obtained from Chapman dataset. As shown in Figure (a) and (c), the representation of ECG signals provided from ST-MEM clusters the lead embedding by limb leads and precordial leads. However, as depicted in Figure (b) and (d), a model that excludes all lead indication modules struggles to cluster ECG representation.
B.7 ADDITIONAL ABLATION STUDY
We extend our analysis through an additional ablation study within our framework. In this study, vary key pre- pre-training stage parameters to understand their impact on the representation. We assess how these changes influence the downstream fine-tuning performance, specifically within the PTB-XI. dataset.
21

---

Published as a conference paper at ICLR 2024
Regular rhythm
Irregular rhythm
(a)
bb)
(4)
Figure 6: A t-SNE plot of ECG signal representation for the regular and irregular rhythm learned from ST-MEM. Each circle represents the single ECG signal representation with different leads. The ellipse with blue and orange indicates the Gaussian (i.e., cluster) obtained from the Gaussian mixture model (GMM). (a) and (c) are our proposed method, ST-MEM, where (b) and (d) are the model that removes all lead indication modules (i.e., excluding shared decoder, lead embedding, and SEP embedding).
Decoder depth. First, we explore the impact of the depth of the wrad-wiee shared decoder. Modifying the depth of decoder can influence the characteristics of the represen- tations learned by the encoder. For instance, if the decoder too shallow, some parts of the encoder might end up tak- ing the role of decoder (Park et al., (022). Table 20 provides insights into the impact of varying the decoder depth on PTB-XL fine-tuning performance. Notably, when configuring the decoder with just a single transformer block, a significant decrease in performance observed. Conversely, excessive depth is also found to be undesirable. The highest performance is achieved with a depth of blocks.
Table 20: Ablation on decoder depth.
Decoder depth
AUROC
0.925
0,933
0.931
11
0.931
Masking ratio. Next, examine the impact of the random masking ratio during pre-training. The masking ratio directly influences the number unmasked patches accessible to the decoder, factor that governs the complexity the reconstruction task. Figure illustrates how PTB-XL fine-tuning performance changes with varying masking ratios. Notably, for ST-MEM, the highest performance achieved moderately high masking ratio of 75%.
0.935
0.930
AUROC
0.925
PTB-XL, Fine-tune
0.920
75
90
Masking ratio (%)
Figure 7: Ablation masking ratio.
Lead indicating modules. We compare the effectiveness of the lead indicating embeddings, SEP and lead embeddings, by removing them one by one. Table 21 the PTB-XL linear evaluation performances of the model trained with without each lead indicating module. The AUROC score the highest when both two modules are hired together, meaning that they help the model to learn spatio-temporal relationships.
22

---

Published as a conference paper at ICLR 2024
Table 21:
Ablation lead indicating modules.
Ablation
AUROC
[SEP] embedding
Lead embedding
X
X
0.822
0.830
0.820
0.838
B.8 ADDITIONAL ANALYSIS OF SELF-ATTENTION MAPS OF ST-MEM
In Figure 8 we demonstrate additional self-attention maps, generated by pre-trained ST-MEM, from ECGs across different rhythm types. It is notable that, regardless of the rhythm types of ECG, ST- MEM consistently assigns high attention weights temporal patches that have similar shapes to query patch and neighboring spatial patches. From this observation, we conjecture that the ST-MEM trained recognize the morphologies ECGs while distinguishing the unique characteristics of each lead, which is essential for general ECG representation learning.
(a)
(
(c)
11
avL
0.6
aVF
0.4
V3
V4
Regubar
Inogular
Imegular
Figure 8: Self-attention maps of ECGs with regular and irregular rhythm. (a) is from regular rhythm ECG, while (b) and (c) are from irregular rhythm ECGs. Pre-trained ST-MEM consistently assigns high attention weights to temporal patches that share similar shape to the query patch.
23

---

Published as a conference paper at ICLR 2024
C FUTURE WORK: APPLYING ST-MEM TO OTHER MULTIVARIATE TIME-SERIES DATA
We believe that methodology be applied to various multivariate time-series domains. An ample this adaptability is the transformation of lead embedding into a general time-series feature embedding. In this section, we demonstrate additional experiment utilizing the Human Activity Recognition (HAR) dataset (Anguita et al. 2013).
HAR dataset is multivariate time series acquired from 30 individuals engaged in daily activities while wearing waist-mounted smartphone equipped with inertial sensors. Data were collected in 2.56 seconds with sampling frequency of 50Hz, resulting 128 readings per sample. The collected data include triaxial acceleration from the accelerometer, estimated triaxial body acceleration, and triaxial angular velocity from the gyroscope, constituting a total of 9 features. The goal to classify six different activities: walking, walking upstairs, walking downstairs, sitting, standing, and lying down.
The experimental settings are follows. The model processes input data with dimensions of 9 128 with a patch size of 32. The encoder consists of blocks with 64 widths and four heads, while the decoder comprises four blocks with 64 widths and four heads. During pre- pre-training, 60% of patches are randomly masked and reconstructed to minimize the mean squared error between raw signals and reconstruction. The pre-training lasts for 200 epochs with fixed learning rate of 0.001. Fine-tuning is performed using cross-entropy for 30 epochs with a fixed learning rate of 0..01.
Table 22 classification results of the fine-tuned model on HAR. ST-MEM shows comparable per- formances, which implies that our proposed model can be extended to the general multi-variate time series domain.
Table 22: Fine-tuning results of human activity classification tasks.
HAR
Methods
Accuracy
F1
Supervised
0.964
0.967
MTAE
0.965
0.968
ST-MEM (Ours)
0.983
0.984
24


Reviews:
[{"review_id": "IswbzRG5Ee", "reviewer": "Reviewer_pLfJ", "summary": "This work proposes a method called ST-MEM (Spatio-Temporal Masked Electrocardiogram Modeling), \nto leverage self-supervised learning (SSL) in order to train a model that can be used for diagnosis of conditions detectable through Electrocardiograms (ECG). \n\nST-MEM is based on a Masked auto-encoder and it uses a vision transformers (ViT) architecture. The 12 leads ECG signal is divided into temporal patches. Some patches are masked and the task is to reconstruct the masked signal. Each encoded patch is then added to a LEAD-specific encoding (a way to identify from which lead the signal is coming from) and to the traditional positional encoding (a way to identify temporary where the patch belongs). Additionally, a special token is appended and postponed to each lead signal. Since some leads share a highly correlated signal the decoder is a Lead-wise shared decoder (i.e. it only attempts to decode the signal from 1 lead at the time, this is to ensure that the task is not trivially solved by copying the masked patched from a highly correlated lead).\n\nAfter training a labeled dataset can be sued to train a linear layer on top of the encoder, or to fully fine-tuned the model.\n\nThe proposed model is pre-trained using a total of 188,480 ECGs 12 leads signals coming from three datasets (Chapman, Ningbo, CODE-15), and if tested using two different datasets PTB-XL and CPSC2018 on the task of detecting cardiac arrhythmia and myocardial infarction. Results (in terms of accuracy, F1 and AUROC) show that with linear fine-tuning, the proposed method performs better than other SSL methods albeit it still underperforms the supervised baseline. When fine-tuning, however, the proposed method surpasses also the supervised baselined in all the metrics employed. \n\nThe current method is also resilient to lower amount of data compared to all alternatives tested as shown by achieving the best results in terms of AUROC using only 1% and 5% of the fine-tuning dataset. \n\nAdditionally, the authors perform experiments using only a sub set of the leads or using only 1 lead (as in the PhysioNet2017 dataset). The proposed technique remains the best  performer across all baselines.", "questions": "Having both the SEP token and the leads embedding seems redundant. Have the authors considered an ablation where only the leads embedding are used but not the SEP? If SEP is needed why is it needed twice and not just at the beginning or just at the end?\n\nHave the author considered using only a subset of the 8 augmentations used for the contrastive SSL baselines shown in Appendix D? Some of them could really alter the signal and be counter productive. For example, the Flip, the shift perhaps also the Sine and partial Sine.\n\nThe initial statement “detecting various heart diseases” seem to imply that the proposed technique could do so, however, the tests only show a couple of heart condition. For example in the PTB XL dataset there are 4 different conditions, why not show the results on all of them?\n\n“The patches undergo flattening” why is flattening needed here? Isn’t the signal already flat?\n\nThe pre-training dataset comes with different sampling rate (two at 500Hz and 1 at 400Hz). How was this taken care of? Was the 500 subsampled? This should be explained.\n\nSimilarly the physionet comes with 200Hz, how was it adapted to the pre-trained model?\n\nFor signals that are longer than 10 seconds, I assume the signal was split into 10 seconds but how was all the outcome computed? Average? Voting? This should be clarified.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper tackles an important problem since high quality ECG labeled data are scarce but ECG data in general is much more available. \n\nThe proposed solution is simple yet very effective as shown in the results, also compared to other SSL alternatives.", "weaknesses": "Some part of the manuscript could be improved. For example:\n\tFigure 2 (c) is not clear. \n\tWhat is “seasonality”?\n\tSee more clarification to add to the manuscript from my questions below\n\nSome experiments could be stronger. For example PTB XL dataset have 4 different conditions but it seems that only a couple were used in the tests.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "WCiq3OI1yU", "reviewer": "Reviewer_knNF", "summary": "This paper presents a self-supervised pre-training method for multi-lead ECG data that can be generalized to reduced lead sets. The method relies on spatio-temporal patch reconstruction, with led-wise shared decoders. The experiments were performed by pre-training on three ECG datasets and fine-tuning to two different down-stream ECG datasets/tasks. Results demonstrated the improvement in downstream tasks in comparison to supervised baselines and various unsupervised pre-training methods.", "questions": "Please clarify the main questions listed in my above comments. \n\nIn addition, it’d be interesting to see Fig 4, the embedding of the ECG signals, across different rhythm types as well, in order to appreciate the clustering across rhythm types versus spatial locations of the lead.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper was overall clearly written and easy to follow.\n\nThe experimentation was thorough, especially in the inclusion of relevant baselines, and the empirical results speak favorably to the contribution of the work.\n\nThe use in low-resource settings is interesting, and the gain brought by proposed pre-training more significant.", "weaknesses": "Despite favorable empirical results, the intuition of the spatiotemporal patching based self-supervised learning is not very clear. It is not clear why such patching will help the learning of spatiotemporal representations. Furthermore, the relation with two baselines — MLAE that reconstructs spatial patches and MTAE that reconstructs temporal patches — need to be better clarified. \n\nThe assumption of the method, in particular to what type of ECG signals and tasks this will apply, needs to be better discussed. For instance, the use of temporal patching seems to rely on the assumption that the rhythm  within the input signal length (e.g., 10s) — whether it is normal or abnormal —is regular and periodic.  Is that true? How would this work for rhythms that are irregular (e.g., atrial fibrillation), or rhythms where the abnormal rhythm only shows up in transient beats within a longer segment (which is typical for some PVCs and tachycardias). If the method is not designed for these scenarios, it should be clearly clarified.  \n\nRelated to the above comments, more details about the data and classification tasks needed to be given. What are the labels of rhythms being classified? Does each 10-s segment only has one label? The paper did a good job in highlighting the it’s important to develop methodology to the application problem at hand — this should also be reflected when describing data and experimental settings, as different ECG tasks can mean that the features one are looking for is very different (e.g. is it a regular rhythm, is it an irregular rhythm, is it a transient rhythm, etc).\n\nOverall, while the performance gain is notable, they are also overall limited — at 2 decimal points compared to supervised baselines wen using 100% data. It is not clear what is the clinical significance of such performance gain. Perhaps it is because the supervised baselines are already quite good in the tasks considered (over 0.9 AUROC in both downstream datasets). It may be more convincing if the authors could find “harder” base tasks in order to see if the proposed method will have clinical significance.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "pxTfveIQ2J", "reviewer": "Reviewer_QUx7", "summary": "This paper proposes ST-MEM (Spatio-Temporal Masked Electrocardiogram Modeling) as an approach to learn a feature representation of ECGs which considered both spatial and temporal aspects of the signal. In particular the authors use a lead-specific decoder to guide the network to learn spatial representation. The author demonstrates the effectiveness of the proposed method on multiple datasets includes PTB-XL, CPSC2018 and Physionet 2017 by comparing to supervised learning baseline, as well as other self-supervised learning methods and get superior results on both general setting and low-resource/reduced lead setting. The author also performs quantitative and qualitative analysis of the captured spatial and temporal relationship, with clustering and attention map respectively.", "questions": "1. In the \"PERFORMANCE IN REDUCED LEAD SETTINGS\" section, 1-lead part was using lead I. How is the choice being made? Would it be valuable to also report 1-lead performance of other 5 leads?\n2. In the same section, all methods seem to have larger 6-lead vs 1-lead gap on PTB-XL than CPSC2018. Does the author have insight on why this is the case?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper is well-written with clear methodology and clarity in details. The presentation is readily comprehensible.\n2. The proposed method has superior performance on important public datasets compared to several other contrastive and generative self-supervised learning methods. The favorable performance of proposed method also extends to practical scenarios of reduced lead and low-resource setting.", "weaknesses": "1. The idea of using MAE to learn representation for ECG signal has been explore previous work(Sawano et al., 2022[1] in the reference). The author should highlight the difference.\n2. The experiment session seems missing some previous works. For example, Temesgen et al., 2022[2] also report metrics on PTB-XL and seems to have higher AUROC score.\n\n\n[1] Shinnosuke Sawano, Satoshi Kodera, Hirotoshi Takeuchi, Issei Sukeda, Susumu Katsushika, and Is- sei Komuro. Masked autoencoder-based self-supervised learning for electrocardiograms to detect left ventricular systolic dysfunction. In NeurIPS 2022 Workshop on Learning from Time Series for Health, 2022.\n[2] Mehari, Temesgen, and Nils Strodthoff. \"Self-supervised representation learning from 12-lead ECG data.\" Computers in biology and medicine 141 (2022): 105114.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "RA0vZWl6LY", "reviewer": "Reviewer_mpnR", "summary": "The authors propose a simple but effective ECG specific generative self-supervised learning framework utilizing both spatial and temporal characteristics of ECG. The project is straightforward, and well examined.", "questions": "To claim \"general ECG representation\", it is quite important what actual task was tested. Could you provide more details of classification of myocardial infarction as well as classification of cardiac arrhythmia, e.g., what are the diagnostic criteria of myocardial infarction? What type of arrhythmia are classified?\n\nThe authors tested Lead I for the single lead task. In a clinical setting, Lead II is the first choice in many situations. Did the author evaluate for Lead II as well?\n\nIt would be quite important for the general audience of ICLR that the authors discuss possible extension of this approach outside of ECG or the medical field.\n\nMinor point.\nIn Figure 2, the authors assigned the same color code to two different segmentations. Could you change one of them to the other color code such as grayscale?", "limitations": "", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "Althogh it's ECG specific, this work provides two important insights. 1) solution for data domain in which cost of labeling is quite high. 2) The electrocardiogram is an indirect observation of the electrical behavior of a single heart from multiple points on the body surface to begin with, and this approach has potential for application to similar data domains that exist not only in medicine.", "weaknesses": "The problem setting is specialized to the ECG, and the scope might be too narrow for the general audiencs of ICLR.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)