Combined self-supervised learning for ECG classification

Authors: doga  gündüzalp
congyu  zou
Venue: IEEE_org_EMBS_BHI_2024_Conference
Published: 2024-06-13 00:13:20

Abstract: Electrocardiogram (ECG) is a crucial non-invasive method for measuring the electrical activities of the heart and detecting cardiovascular diseases. While deep learning approaches for cardiovascular disease classification have gained popularity, creating labeled data remains expensive. Contrastive and generative self-supervised learning methods, particularly with the vision transformer (ViT) neural network architecture, have been introduced as leading solutions to this challenge. Although the [CLS] token, which is one of the most important components of ViT, is frequently utilized for aggregate representation in supervised learning scenarios, such as diagnosis, its exploitation in self-supervised tasks has not been extensively explored. This study proposed a method to incorporate multiple pretraining tasks for better representation learning via utilizing the [CLS] token more effectively. Based on this method, we introduced two novel combined self-supervised learning frameworks for ECG analysis, namely MAE-MoCo and MAE-Nextclip. The MAE-MoCo framework combines generative and contrastive self-supervised learning by incorporating a masked autoencoder with a momentum encoder. On the other hand, MAE-Nextclip is a generative method that reconstructs not only masked patches but also Nextclip data with the assistance of the [CLS] token. We validated our methods on a joint database of China Physiological Signal Challenge in 2018, Physikalisch-Technische Bundesanstalt XL, and Chapman datasets. The fine-tuned models outperformed the state-of-the-art models in the Chapman dataset with macro-F1 of 0.96 and the area under the curve of receiver operating characteristic of 0.99. The outstanding performance on downstream tasks demonstrates the potential of combining pretraining tasks, especially generative and contrastive tasks, in the field of automatic ECG interpretation.

Main paper: 
 i. introduction: 
 Electrocardiogram (ECG) is a non-invasive method that records activities by measuring the electrical signals of the heart [1] . Diseases related to the heart, also known as cardiovascular diseases, hold significant importance among causes of death [2] . ECG stands as one of the most effective and prevalent tools used in the diagnosis of cardiovascular diseases due to its ability to promptly generate results, non-invasiveness, and capability to measure the activities of the heart, the most vital organ. Due to probes attached to different points of the body, ECG signals consisting of 12 leads encapsulate the electrical activity of the heart and this is done for a certain period of time. Due to these attributes, ECG signals can be treated both as a time series and as a two-dimensional medical image.ECG data has been identified as a compelling area of study within the field of machine learning (ML) research. ML models have gained significant value in recent years, showcasing impressive performances in various fields such as computer vision (CV) [3] , natural language processing (NLP) [4] , [5] . The successful operation of non-linear algorithms with ECG signals signifies the potential effectiveness of ML-based models in the field of ECG. The performance of traditional ML algorithms such as Gaussian Naive Bayes and random forest stands out in heart disease classification [6] , while the impact of deep learning models in arrhythmia beat classification has also been demonstrated by numerous studies [7] . Especially from the CV perspective, convolutional neural network (CNN) based architectures such as ResNet [8] , InceptionTime [9] and from the time series perspective, recurent neural network (RNN) based long short-term memory (LSTM) serve as highly important backbone candidates [10] , [11] .Supervised learning in ECG analysis requires extensive labeled data, often necessitating significant time and expertise, particularly from cardiologists, to label such medical data accurately. Due to the widespread use of ECGs, large volumes of unlabeled data are available, yet labeling these for supervised learning is labor-intensive. In contrast, selfsupervised learning leverages unlabeled data, enabling models to generate their own labels, thus addressing the limitations of supervised learning. Self-supervised learning is particularly beneficial when labeled data is scarce and may outperform the models trained solely on supervised data [12] . Self-supervised learning methods, categorized into generative, contrastive, and adversarial learning [13] , produce pretrained encoders that serve as robust feature extractors for downstream tasks, highlighting the growing importance of self-supervised learning in ECG analysis.In the field of ECG, there exist frameworks derived from the two primary branches of self-supervised learning, namely contrastive and generative learning perspectives, demonstrating exceptional performance. Within the contrastive learning perspective, it has been shown that simple contrastive methods, including end-to-end trained SimCLR [14] are effective.Additionally, models like MoCo [15] and BYOL [16] , which use gradient descent for generating query and key representations in one branch and momentum updates in another, also demonstrate significant performance. Although SimCLR, MoCo, and BYOL are effective contrastive self-supervised learning methods, none of them are designed specifically for ECG. One of the most effective contrastive learning methods designed specifically for ECG is ISL, as it is tailored for multivariate cardiac signals [17] . This advanced contrastive learning method combines intra-subject and inter-subject selfsupervised techniques. The intra-subject approach examines temporal dependencies in ECG signals from the same patient, while the inter-subject approach computes contrastive loss by treating augmented signals from the same patient as positive pairs and signals from different patients as negative pairs.From the generative learning perspective, the models presented in the ECG domain are primarily based on masked autoencoding (MAE) [18] . Models such as masked time autoencoder (MTAE) [19] , which is pretrained to find the masked ECG patches in the time axis, masked lead autoencoder (MLAE) [19] , which reconstructs masked patches along the lead axis, are among the state-of-the-art performing models utilizing masking in the ECG classification domain. In addition, MassMIB, another method based on masking technique, encompasses an encoder-decoder pair that encodes a signal masked in the time domain and reconstructs the masked tokens in time, alongside an additional encoder-decoder pair that performs the same task in the frequency domain. Lastly, Nextclip [20] reconstructs the next semi-cardiac cycle by removing ECG data after any given time point as a pretraining task.Recent advancements in contrastive self-supervised learning for ECG data, such as SimCLR, MoCo, and BYOL, have been significant but the rise of transformer-based architectures coupled with methods like MAE has led to state-of-the-art performance in generative self-supervised learning methods. The backbone architecture of the encoder is crucial for performance, with vision transformer (ViT) outperforming ResNet [21] in many ECG-specific pretraining tasks [19] . A notable feature of ViT is the inclusion of the [CLS] token, which, as seen in BERT, is beneficial for aggregate representation. However, in ECG pretraining tasks, the [CLS] token does not leverage its characteristic of carrying aggregate representation effectively, suggesting the potential for more optimal approaches in pretraining.This paper introduces a novel approach that combines pretext tasks in generative and contrastive learning perspective using the [CLS] token to enhance ECG signal representation. The primary objective is to create a robust pretraining task by incorporating the [CLS] token of ViT into a hybrid learning framework for ECG signal representation. In this context, two approaches are explored: adding a contrastive projection head to the [CLS] token output of a MAE framework using 1D-ViT called as MAE-MoCo, and incorporating a new Nextclip decoder to create a combined generative self-supervised learning framework named as MAE-Nextclip. MAE-MoCo aims to develop an encoder that targets both the reconstruction of the ECG signal and the differentiation of positive and negative pairs, while MAE-Nextclip reconstructs the masked patches as well as the ECG data in the next semi-cardiac cycle, outperforming current state-of-the-art techniques. In addition, the novel loss functions tailored to these combined pretraining tasks are introduced. The experimental results highlight the effectiveness of the proposed MAE-MoCo and MAE-Nextclip frameworks, demonstrating their potential to set new benchmarks in self-supervised learning methods for ECG signal analysis.The proposed methodology includes two novel pretraining techniques, namely MAE-MoCo and MAE-Nextclip, which are later fine-tunable on specific datasets. Both models effectively utilize the [CLS] token of ViT in their pretraining tasks. The MAE-MoCo model merges generative and contrastive learning perspectives, whereas MAE-Nextclip combines two distinct generative learning tasks. The encoder is based on a 1D-ViT structure and comprises two parts for MAE and MoCo. The MAE part of the encoder takes the original ECG data without any data augmentation and first passes it through a patch embedding block. This block, which is a 1D CNN block with kernel size and stride equal to the patch size, divides the ECG data into patches of a specified size and maps each patch to the hidden size. Next, positional embedding is added to each patch to ensure that positional information is not lost. Although a fixed sinecosine embedding is used for positional embedding in [18] , this study opts for learnable positional embedding parameters since the encoder is used for both MAE and MoCo. Following this, a certain percentage of patches are masked using random masking. It should be noted that, only the unmasked patches are used afterwards. Finally, the masked encoded ECG signal is obtained by passing through six consecutive stacked transformer blocks and a layer normalization block. The purpose of the decoder network is to reconstruct the masked parts with the least error by replacing them with learnable parameters based on the encoded output for generative learning (x generative ) in the encoder. The decoder block takes the x generative and first passes it through a patch embedding block. This block is structurally different from the encoder's patch embedding block because the decoder does not perform patch separation; instead, patches are mapped to a different dimension using a linear layer. Subsequently, a learnable [mask] token is placed at the position of the masked patches resulting from random masking in the encoder, and positional embedding is added to each patch. Next, similar to the encoder, the decoder employs transformer and layer normalization blocks. Finally, another linear layer is used to match the output dimension with the original ECG, aiming to reconstruct the masked ECG patches with minimum error. The overall structure of the decoder is given in Fig. 3 .The momentum encoder in the MAE-MoCo framework uses the same architecture but ignores the output x generative . Additionally, the data augmentation technique used is different from that in the encoder. Similar to MoCo, the parameters of the momentum encoder are trained with the momentum update of the encoder, not with the gradient descent. With each batch, the oldest batch in the queue is removed, and a new batch is added, following the principles of MoCo training. A summarized working scheme of the MAE-MoCo framework is given in Fig. 4 and detailed hyperparameters of the encoder, decoder, and momentum encoder are described in Table I , II and III, respectively. The architectural hyperparameters of the momentum encoder are exactly the same as the encoder. 2) MAE-Nextclip: The MAE-Nextclip framework consists of a shared encoder and two separate decoders for the tasks of MAE and Nextclip reconstruction, respectively. Architecturally and functionally, this model bears significant resemblance to the MAE-MoCo model.The MAE component of the encoder is identical to that described in Section II-A1. For the Nextclip part, the original ECG data is first transformed into Nextclip data. In this context, a random starting point is selected within a patch of the ECG data. A semi-cardiac cycle is then extracted from this randomly selected point and used as the ground truth data, and the ECG data from this point onward is set to zero for use in the rest of the encoder. The prepared ECG data for Nextclip is embedded with patch embedding and positional encoding. Additionally, a [CLS] token is appended for use in the Nextclip decoder. Finally, the resulting signal undergoes encoding through stacked transformer blocks and layer normalization. The operational schematic of MAE-Nextclip encoder is provided in Fig. 5 . The MAE decoder is identical to the one depicted in Fig. 3 . However, the Nextclip decoder initially extracts the [CLS] information from the x generative,N extclip signal and then proceeds with patch embedding. Afterward, without introducing a [mask] token, it sequentially traverses through transformer, layer normalization, and linear layer to reconstruct the Nextclip signal. A summary of the decoders used separately for MAE and Nextclip is provided in Fig. 6 .The overall block diagram of the MAE-Nextclip model is given in Fig. 7 . Since the hyperparameters of the MAE-Nextclip model exhibit substantial similarities with those of IV . Furthermore, the logic behind the selection of the 9 in the size matcher network is rooted in the attempt to predict a cardiac cycle of length 9 patches, starting from where the data is cropped during the creation of Nextclip data. The pretraining and fine-tuning of the models utilized an Intel(R) Core(TM) i7-11800H @ 2.30GHz CPU, 16GB RAM, and an NVIDIA GeForce RTX 3060 GPU.1) Pretraining: The pretraining of the models utilized a combination of three different datasets namely China Physiological Signal Challenge in 2018 (CPSC2018) [22] , Physikalisch-Technische Bundesanstalt XL (PTB-XL) [23] , and Chapman [24] . Due to the high number of ECG samples and the diversity of data collected from various hospitals, the dataset exhibits significant diversity. Therefore, it is highly valuable for pretraining.In order to enhance the stability of the outputs obtained from the models with the provided input data, the inputs are Z-score normalized. In other words, the mean of the input data is subtracted and divided by its standard deviation.EQUATIONDue to the composition of the MAE-MoCo and MAE-Nextclip models from two different pretraining techniques, conventional loss functions cannot be directly utilized. Instead, a combination of these loss functions is employed. The MAE-MoCo model comprises a combination of generative and contrastive loss functions. The generative loss function calculates the MSE between the original data and the reconstructed data of masked patches. The generative loss is illustrated by (2) [19] .EQUATIONFor the contrastive loss, the instance discrimination method is utilized. Thus, if the encoded keys and queries originate from the same ECG, they are considered as positive pairs; otherwise, they are treated as negative pairs. The contrastive loss is mathematically described in (5) [15] , where ℓ pos and ℓ neg represent the positive and negative logits, respectively, [] denotes concatenation operation, and CE stands for crossentropy loss function.EQUATIONEQUATIONAfter the computation of the generative and contrastive losses, these two loss functions are combined for the MAE-MoCo model as depicted in (6) .EQUATIONSimilarly, the MAE-Nextclip model employs (2) , where the variable x reconstructed,i is substituted with the output x reconstructed,M AE of the MAE-Nextclip model. Since the MAE-Nextclip model is a combination of two different generative methods, it utilizes the Nextclip loss instead of the contrastive loss. As demonstrated in (7) , the Nextclip loss is computed as the MSE between the reconstructed cardiac cycle and the ground truth. The overall MAE-Nextclip loss function is illustrated in (8) .EQUATIONThe α parameter in the loss function of both models is a hyperparameter and experiments are conducted with different α values. However, in the initial case, the α is set to 40 for MAE-MoCo and 2 for MAE-Nextclip. The reason for this is to bring both losses to the same scale. Additionally, training has been conducted for MAE-MoCo with α values of 20 and 10, while for MAE-Nextclip, configurations with α values of 1 and 0.5 have been employed.Both models update their own weights to minimize their respective combined batch-wise losses during training. Warmup epochs are applied by linearly increasing the learning rate for the initial specified number of epochs. Warm-up epochs serve to stabilize the training process and accelerate convergence [25] . Throughout the training process, the learning rate is reduced as epochs progress with the aid of learning rate schedulers. If there is no improvement in the validation loss after a certain number of epochs, the learning rate decreases by a specified rate. The detailed configuration of the models used for pretraining is provided in Table V . It should be noted that in addition to the hyperparameters and pretraining configuration parameters of models, there are certain fixed parameters used in models as well. The length of the ECG data is 3600, with 12 channels. The length of patches used by ViT is chosen as 25 and 75% of the patches are masked in MAE. Therefore, the models have 144 tokens, excluding the [CLS] token. The fixed parameters are listed in Table VI . Training of MAE-MoCo and MAE-Nextclip take approximately 77 hours and 42 hours, respectively. Additionally, while MAE-MoCo has 27M parameters, the number of trainable parameters in MAE-Nextclip is 15M.2) Fine-tuning: The pretrained models, using all three datasets mentioned in Section II-B1, are fine-tuned on Chapman dataset. After pretraining, the encoders of the models are retained, while the remaining components (momentum encoder and decoder for MAE-MoCo, two decoders for MAE-Nextclip) are discarded. It should also be noted that the projection head of the MAE-MoCo encoder is not utilized for fine-tuning purposes.A linear network with a sufficient number of output nodes capable of classifying the appropriate dataset is connected to the [CLS] tokens of the pretrained encoders. The resulting architecture is tuned using two distinct techniques, namely the linear probe and fine-tuning methods. In the linear probe approach, all trainable parameters of the pretrained encoder are frozen, and only the parameters of the new linear network connected to the [CLS] token are updated during task-specific training. Consequently, in this mode, the success of pretraining is more accurately assessed since the encoder functions as a feature extractor, while the linear layer acts as a linear classifier. Conversely, in the fine-tuning mode, the encoder is initialized with pretrained parameters, but both the encoder and linear layer parameters are trained end-to-end during taskspecific training.The objective of the models in fine-tuning is to minimize the cross-entropy loss between the one-hot encoded ground truth category labels and the probabilities outputted by the model. In fine-tuning, as in pretraining, warm-up epochs and learning rate schedulers are utilized for similar purposes. Additionally, early stopping is employed during fine-tuning. If no improvement in validation loss is observed over a certain number of epochs, the fine-tuning training is terminated. The model with the best performance in terms of validation loss is selected and used for testing. The detailed configuration of the fine-tuning is provided in Table VII Accuracy is a metric commonly encountered in nearly all classification tasks, shedding light on the model's classification ability. Accuracy is calculated by the ratio of correctly predicted individual samples to the total number of test samples [26] .It is noteworthy that the calculation of accuracy does not account for any imbalance in the data distribution among the classes in the test dataset. Thus, macro-F1 score and area under curve of receiver operating characteristic (AUC) come into play. The macro-F1 score is computed as the harmonic mean of precision and recall [26] . Precision measures the classifier's ability to not misclassify negative examples as positive, while recall represents the ability to find all positive examples [26] .AUC, on the other hand, is a metric that demonstrates the classifier's performance at different thresholds and aims to measure how well positive and negative examples are separated. AUC is a metric defined for binary classifiers; hence, in this study, since the classifiers are multi-label classifiers, one-vs-one AUC is employed. One-vs-one AUC computes the AUC metric by averaging over all possible class pairs.The proposed models need to be compared with previous and state-of-the-art self-supervised learning methods in terms of metrics to demonstrate the effectiveness of the models. Metrics commonly used in model comparison, such as accuracy, macro-F1 score, and AUC, are utilized in comparison. The MAE-MoCo and MAE-Nextclip models fine-tuned with Chapman dataset are initially compared with the state-ofthe-art on published benchmark that are specific to the field of ECG. Subsequently, MAE-MoCo and MAE-Nextclip are compared with previous prominent self-supervised learning methods such as MTAE [19] , Nextclip [20] , MoCo [15] , and SimCLR [14] .MAE-MoCo and MAE-Nextclip are compared with Mass-MIB [27] , a leading method in generative self-supervised learning demonstrating state-of-the-art performance on Chapman dataset, and ISL [17] , representing the contrastive learning aspect in the field of ECG. Comparison results on the Chapman dataset is presented in Tables VIII. MassMIB and ISL utilize the higher-level classes of the Chapman dataset; hence, the comparisons are conducted with 4 classes, and the AUC metric is included. The configuration of Chapman dataset with 4 classes is called as Chapman-Reduced for clearance. When comparing the results, it is observed that in the Chapman dataset, the proposed models have outperformed both MassMIB framework using fine-tuning technique in terms of macro-F1 by 0.02 and ISL models in terms of AUC by 0.03 in linear probing and by 0.007 points in fine-tuning. MassMIB employs masking in both time and frequency domains to discover generative features, while ISL employs intra-patient and inter-patient discrimination to achieve contrastive features. In proposed models, the extraction of both contrastive and generative features is the most significant factor in outperforming ISL and MassMIB. Furthermore, it is observed that the MAE-Nextclip model performs better than MAE-MoCo in the Chapman dataset.One possible reason behind this is that one of the most important features of the Nextclip model is its successful detection of atrial fibrillation. Since a significant portion of the Chapman dataset consists of atrial fibrillation, MAE-Nextclip outperforms MAE-MoCo in terms of metrics.MAE-MoCo and MAE-Nextclip demonstrated robust and effective performance when compared to state-of-the-art on published benchmarks, surpassing the compared methods on Chapman dataset. Moreover, the metric comparisons of MAE-MoCo and MAE-Nextclip models that are pretrained with CPSC2018, PTB-XL, and Chapman datasets, with MTAE [19] , Nextclip [20] , MoCo [15] , and SimCLR [14] utilizing linear probe and fine-tuning techniques on Chapman dataset, are presented in Table IX . The compared models are implemented from scratch with the configuration used by MAE-Nextclip and MAE-MoCo for a fair comparison. It should be noted that in this comparison, unlike some literature like MassMIB and ISL where classes are merged into a higher-level class to reduce the number of classes to 4, the original Chapman dataset is utilized without merging the labels. Due to the large number of classes in the Chapman dataset, AUC is not used for comparison. Upon examination of Table IX , it is evident that MAE-MoCo and MAE-Nextclip outperform previous methods. The proposed models exhibit a noticeably superior performance across all metrics compared to other self-supervised learning methods. This is primarily attributed to the optimization of two different tasks during pretraining. Although the resulting models from pretraining are capable of performing two tasks rather than excelling in a single task, the individual tasks operate with slightly lower performance. The proposed models achieve a balanced performance between these tasks. Therefore, the learned representations are less task-specific and more universal, thereby preserving their potential for fine-tuning. Consequently, when these pretrained encoders are used in finetuning, they approach the performance of optimal classifiers more closely.For different α values, it is seen that the MAE-MoCo performs better, especially with lower α values, due to its effective regularization against overfitting in fine-tuning mode, particularly on the smaller dataset such as Chapman dataset. For MAE-Nextclip, the model improves performance with α = 1 due to stronger masking features, as opposed to moving overfitting region when α increased from 0.5 to 1.The MAE-MoCo and MAE-Nextclip frameworks are indeed powerful models that outperform other models, but they have some limitations such as high computational costs and the challenge of finding optimal parameter sets.Due to the fact that MAE-MoCo and MAE-Nextclip frameworks consist of a combination of two different pretext tasks, the computational cost required for training is higher compared to individual pretext tasks. Specifically, contrastive models have even higher costs compared to generative models due to the double forward pass. Consequently, MAE-MoCo and MAE-Nextclip can be considered more costly than other models.Another limitation of the proposed frameworks is the increase in the number of non-trainable parameters resulting from the combination of two different models. The increase in the number of non-trainable parameters complicates the task of finding the optimal parameter set compared to other individual models.To further enhance the performance of the proposed frameworks, future work requires the utilization of different, more advanced backbones, the combination of different tasks, the utilization of different types of data to test models and more hyperparameter tuning.The convolutional vision transformer (CvT), incorporating convolutions into the ViT architecture [28] , has proven its efficacy on datasets like ImageNet and different backbones like CvT, ResNet could serve as a promising candidate for performing combined tasks more effectively in the ECG domain.Additionally, the combined tasks can be altered or enhanced, with alternative tasks such as Centerclip which reconstructs randomly selected segments of ECG data rather than the next segment, SimCLR, BYOL, DINO [29] . Moreover, another strategy to improve the performance of combined selfsupervised learning tasks could be increasing the number of tasks.Furthermore, experimenting the models with other physiological signals that have ECG signal characteristics such as being a time series and a 2D image can better demonstrate the effect of the combination strategy.Finally, more optimal fixed parameters can be found with more computational power since MAE-MoCo and MAE-Nextclip have many fixed parameters. Although the values used are the most optimal ones in the original frameworks, they may not be the most ideal values for MAE-MoCo and MAE-Nextclip, so the proposed models may have more optimized versions.In this paper, novel pretraining tasks are introduced from the perspective of self-supervised learning, arising from the significant problems of obtaining labeled data. The MAE-MoCo framework is proposed to combine the strengths of generative self-supervised learning, exemplified by MAE, and contrastive learning, as seen in MoCo, thus allowing for the simultaneous utilization of generative and contrastive learning. Additionally, MAE-Nextclip emerges from the merge of two distinct generative learning tasks, MAE and Nextclip. The fundamental factor contributing to the emergence of both models is the [CLS] token within ViT which serves as an aggregate representation. While the MAE task inherently operates at the level of individual patch tokens to reconstruct masked patches, the utilization of information from the [CLS] token suffices for Nextclip and MoCo tasks.The fine-tuning results demonstrate that the proposed models are achieving new state-of-the-art results on the Chapman dataset. The primary reason for outperforming other models lies in the concurrent extraction of generative and contrastive features with the assistance of the pretrained encoder in the proposed models. 
 
 Appendix:  



Reviews: review: title: title: The paper proposes the idea of combined self-supervised learning for ECG classification
paper_summary: paper_summary: The paper proposes the idea of constrictive and generative self-supervised learning methods such as vision learning transformer (ViT) neural network architecture to resist the dependency on expensive labeled data for cardiovascular disease classification.
strength_weakness: strength_weakness: 1. The Table VIII. and IX seems bit confusing and saturated, The authors are requested to enhance the quality of visibility and make these more clear.
questions: questions: 1. The authors are requested to justify the source of equation (9), (10),(11) and (12).
title: title: ECG self-supervised learning for classification, MAE-MoCo and MAE-Nextclip
paper_summary: paper_summary: This paper introduces two novel self-supervised learning frameworks, MAE-MoCo and MAE-Nextclip, for ECG classification. The frameworks leverage a Vision Transformer (ViT) architecture and incorporate the [CLS] token to improve representation learning. MAE-MoCo combines generative and contrastive learning using a masked autoencoder (MAE) and a momentum encoder (MoCo). MAE-Nextclip enhances generative self-supervised learning by reconstructing both masked patches and Nextclip data. The methods are validated on a joint database, including CPSC2018, PTB-XL, and Chapman datasets.
strength_weakness: strength_weakness: 1. Core concept communication and unclear title: The title "Combined self-supervised learning for ECG classification" is somewhat vague. It is not immediately clear what methods are being combined. The core idea of using the [CLS] token as an aggregate representation should be better emphasized and clearly conveyed in both the title and the body of the paper.
2. High computational cost, and the complexity in hyperparameter tuning: The combination of multiple pretext tasks increases the computational burden, making the models less accessible for researchers with limited resources. The models (with Transformer backbones) have a large number of hyperparameters, which may require extensive tuning to achieve optimal performance. This complexity could be a barrier to reproduction and application in other settings.
3. (minor issue) Generalization beyond ECG data: The applicability of the proposed methods to other types of physiological signals or medical data is not explored, which could limit the impact of the work.
questions: questions: 1. Clarity of Title and Core Concepts: The term "combined self-supervised learning" in the title is unclear (what exactly is being combined): more specifically, the core idea of leveraging generative+contrastive SSL, as well as using the [CLS] token as an aggregate representation. Could you clarify and better convey these points in the title and body of the paper? Impact: Clarifying this with some additional text could enhance the paper's readability and ensure the core concepts are well-understood.
2. Could you provide more details on the computational resources required for training the proposed models?How sensitive are the models to changes in hyperparameters (especially the parameters for Transformer architecture)? This would help assess the feasibility for other researchers. Impact: Clarifying the model size and/or FLOPS could influence the evaluation of the model's practicality and accessibility.
3. Hyperparameter Sensitivity: How sensitive are the models to changes in hyperparameters? Have you performed any sensitivity analysis? Impact: Understanding hyperparameter sensitivity would provide insights into the robustness of the models and their ease of use.
title: title: A technically solid paper, but might need further evaluation and clarification
paper_summary: paper_summary: This paper proposes a novel approach to address the challenge of limited labeled data in ECG classification by combining generative and contrastive self-supervised learning methods. The authors introduce two variants of the framework, MAE-MoCo and MAE-Nextclip, which leverage the Vision Transformer (ViT) as backbones. MAE-MoCo combines a masked autoencoder with a momentum encoder for generative and contrastive learning, while MAE-Nextclip combines two generative learning tasks, masked autoencoding and Nextclip reconstruction. The models are pre-trained on a large unlabeled ECG dataset and then fine-tuned on a smaller labeled dataset for classification. The results demonstrate that the proposed models outperform state-of-the-art self-supervised learning methods on the Chapman dataset, highlighting the potential of combining pre-training tasks for improved ECG signal representation and classification.
strength_weakness: strength_weakness: - The evaluation is primarily conducted on the Chapman dataset. Further validation on other publicly available ECG datasets would strengthen the generalizability of the findings.
- Lack some necessary ablations. The choice backbones might need an comprehensive ablation for reasoning. As indicated in some previous ECG-based deep learning studies, the convolutional architecture might still outperform the ViTs in processing ECG data. I am wondering how will the performances be if the the authors use CNN structures, such as ResNet, as the backbone encoder.
questions: questions: - How is the computational complexity of the proposed method? How is the pretraining time compared to other SSL methods such as SimCLR or BYOL?
- I might missed it somewhere, but what is the performances of purely supervised method? The tables only covers the benchmarks from SSL methods.