C-MELT: Contrastive Enhanced Masked Auto-Encoders for ECG-Language Pre-Training

Authors: Manh Pham Hung, Aaqib Saeed, Dong Ma
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords: Multi-modal Representation Learning, Contrastive Masked Auto-Encoders, ECG-Text Pre-Training

Abstract:
Accurate interpretation of Electrocardiogram (ECG) signals is pivotal for diagnosing cardiovascular diseases. Integrating ECG signals with their accompanying textual reports holds immense potential to enhance clinical diagnostics through the combination of physiological data and qualitative insights. However, this integration faces significant challenges due to inherent modality disparities and the scarcity of labeled data for robust cross-modal learning. To address these obstacles, we propose C-MELT, a novel framework that pre-trains ECG and text data using a contrastive masked auto-encoder architecture. C-MELT uniquely combines the strengths of generative with enhanced discriminative capabilities to achieve robust cross-modal representations. This is accomplished through masked modality modeling, specialized loss functions, and an improved negative sampling strategy tailored for cross-modal alignment. Extensive experiments on five public datasets across diverse downstream tasks demonstrate that C-MELT significantly outperforms existing methods, achieving an average AUC improvement of 15% in linear probing with only one percent of training data and 2% in zero-shot performance without requiring training data over state-of-the-art models. These results highlight the effectiveness of C-MELT, underscoring its potential to advance automated clinical diagnostics through multi-modal representations.

Decision: reject
Reviewer scores: 6, 5, 8, 3

Reviews and discussion:
<thread>
creation date:
1730047319851

thread_type:
official_review

<thread_object>
creator: reviewer_jws3

<content>
summary:
This work proposes a multimodal ECG learning framework with multiple objective alignments and evaluates it across various downstream tasks.

strengths:
Many empirical studies encompassing a wide range of datasets and methods.

weaknesses:
- Lack of novelty: MEM, ETM, and MLM losses are very similar to the BLIP [1] work, which was proposed in 2022. This work directly reimplements the BLIP loss in the ECG domain, which is not novel enough.
- Ambiguous results: In Table 3, C-MELT claims that the zero-shot classification performance across all downstream datasets is **77.71**, which is higher than MERL [2]. However, in the ablation results Table 7, the average zero-shot result is **72.5±9.1**, which *is not consistent* with the author-reported performance. Additionally, in MERL [2]'s original paper, the reported average zero-shot performance is **75.24±1.7** in Tables 5-9, which is much higher than C-MELT's performance.
- The reproducibility concern is made worse by the fact the authors don't appear willing to share their code.

[1] Li, Junnan, et al. "Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation." International conference on machine learning. PMLR, 2022.
[2] Liu, Che, et al. "Zero-Shot ECG Classification with Multimodal Learning and Test-time Clinical Knowledge Enhancement." Forty-first International Conference on Machine Learning.

questions:
- Can the authors clarify the main differences between C-MELT and BLIP? Even though the data is different (ECG vs. Image), the framework and optimization objectives appear too similar.
- How do the authors explain the discrepancy between the reported results in Table 3 and Table 7?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730323885771

thread_type:
official_review

<thread_object>
creator: reviewer_vcwp

<content>
summary:
The present manuscript proposes multiple interesting ideas for enhancing ECG-language pre-training by combining contrastive learning and reconstruction.  By doing so, considerable improvements over unimodal pretraining and a recent multimodal approach are shown on multiple downstream datasets.

strengths:
Strengths of the manuscript include the introduction of multiple novel ideas, in particular the combination of learning via contrastive and generative generation. Furthermore, strong results are obtained across datasets. The authors clearly explain the various parts of the approach.

weaknesses:
The manuscript could benefit from a clearer rationale behind the choice of methods, as theoretical justifications are somewhat limited. While several ideas are presented, a more thorough exploration of the reasoning behind each would strengthen the approach. Additionally, although some ablation analyses have been performed, it would be helpful to clarify which components (e.g., reconstruction vs. no reconstruction) play an essential role in the downstream tasks.

questions:
For the experiments:
-	Whereas the proposition to combine reconstruction and contrastive learning is very worthwhile, to my understanding this matter is not directly addressed in the paper. While the Siglep loss is ablated, I believe in this case the model still is not trained solely via reconstruction, as ECG-Text Matching (ETM) is active. I believe the manuscript would be considerably improved if results could be shown under a setting of ‘reconstruction only’ and ‘without reconstruction’. Ideally, under the ‘reconstruction only’ setting, additional information is provided about the utility of text or ECG reconstruction. In the ‘without reconstruction’ setting, it would be interesting to ablate either the Siglep loss and/or ETM. The authors are free to provide alternative analyses than specifically those considered here, but again, gaining an understanding of the contribution of the individual parts of the approach would be important.
-	Why did the authors choose not to compare against other methods for ECG-text pretraining such as the cited Lalam et al. 2023 or Liu et al. 2023 (https://arxiv.org/pdf/2309.07145)?

In the text, the following sections could improve with clearer justifications of the authors work:
- Lines 60-68: Here the authors propose to combine contrastive and generative approaches to leverage their complementary strengths, but it is unclear what these complementary strengths are expected to be. As no clear argumentation is provided why these approaches should be combined, there is no framework in place to later interpret the results. I would therefore recommend to provide a more precise description of the working hypothesis.
-	Lines 240-243: “This can hinder the model’s capability to learn discriminative features (…)”. What is being referred to here? The fact that generative approaches perform reconstruction and contrastive learning distinguishes between data pairs is, to my understanding, precisely what causes them to learn discriminative features to the extent that they do. 
Next, it is suspected that ETM could serve as a contrastive loss but that it may be insufficient as it performs binary classification with fused features. As these characteristics of ETM were introduced with little justification, one is left wondering why the Siglep loss and ETM are both needed. Specifically, the hypothesis that alignment of both unfused and fused features is beneficial is currently not answered in the paper, as ablations do not seem to include ETM.
-	Lines 274-275: I would like to ask why the authors claim nearest-neighbour negative sampling makes negative samples challenging? Does the approach not make negative samples easier by selecting less similar reports as negatives? (And as reports with high cosine distance are selected, is this not opposite to the principle of ‘nearest-neighbours’?)
</content>
</thread_object>

</thread>
<thread>
creation date:
1730546744239

thread_type:
official_review

<thread_object>
creator: reviewer_2pj1

<content>
summary:
This paper introduces C-MELT, a cross-modal pre-training framework for self-supervised learning of ECG signals and textual reports. C-MELT combines the strengths of generative and contrastive learning through masked language and ECG modeling to reconstruct data. It also leverages contrastive loss functions and a nearest-neighbor negative sampling strategy to improve cross-modal alignment. Extensive experiments across multiple public datasets demonstrate that C-MELT significantly outperforms existing methods, especially in downstream tasks like zero-shot learning and linear probing, showcasing strong generalization capabilities and diagnostic potential.

strengths:
- Overall, this paper makes clear contributions, especially the comparative interpretation with MERL, which further promotes the development of multimodal ECG research.
- N3S provides an effective negative sample selection strategy to effectively select negative text reports in medical datasets.

weaknesses:
- The ablation experiment does not discuss the role of different components in the reconstruction method. The contrast method is currently the main method for multimodal alignment in ECG multimodal researches. This paper proposes a hybrid method to enhance the performance, but does not discuss the ablation experiment of the added reconstruction method in more detail, which leads to doubts about the effectiveness of the added reconstruction method.
- The layout of Table 2 is somewhat inadequate, as it differs significantly from the original table in MERL, leading to some confusion when reading. Given that this table may serve as a benchmark in the future, it should be adjusted for consistency to facilitate easier comparison.

questions:
- Please provide more details on the performance improvement achieved by the reconstruction approach.
- In Section 3.1, positional encoding is added at the end. How is sequence information ensured during the calculation of multi-head attention according to the content of the article?
- If ECG-Text Matching (ETM) promotes the alignment of corresponding ECG-Text pairs, why does it not enhance the discriminative ability of the encoders?
- Conversely, if Siglep Loss can accomplish contrastive learning for multimodal alignment, what is the purpose of retaining ETM?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730634239369

thread_type:
official_review

<thread_object>
creator: reviewer_xh2s

<content>
summary:
The proposed approach aims to integrate generative pre-training with robust cross-modal representation learning. The paper extends the previous MERL[1] methodology developed for multimodal learning on ECG records and associated reports, providing zero-shot classification capabilities. The main contributions are the integration of predicting masked segments for both ECG signal and textual reports, specialized loss functions, and an improved negative sampling strategy for cross-modal alignment. The performance is evaluated for diverse datasets and demonstrates superior performance to existing SSL (Self-supervised learning) approaches.

strengths:
The strength of the model is the ability to provide improved performance compared to MERL[1], achieved through the specialized loss functions.

weaknesses:
The method may not be directly comparable to SSL methodologies as the pretraining is exposed to cardiac diagnostic texts enhancing the performance for related tasks. Performance for novel features, unrelated to the context of the reports (e.g. age and sex of the subject) may be impacted adversely.

questions:
Methodology:

Please clearly list the main contributions of the current work in the context of the previous literature, with an accurate formulation of the specific problem addressed.

Please explicitly state whether the primary contribution is zero-shot classification or learning generalized ECG features, and how your work advances the field.

The introduction mentions the scarcity of labeled data but integrating textual reports cannot be regarded as truly “unlabelled” since converting textual descriptions to labels is a trivial task, looking at the examples provided. Please clarify the definition of "unlabeled data" in this context and discuss how the approach differs from traditional supervised learning using text-derived labels.
 
Please categorize your work as self-supervised, semi-supervised, or supervised representation learning and explain why it fits that category, given the use of textual reports.

Does the implementation necessitate multi-modal data or can it also incorporate a mix of ECG-only and multi-modal data? 

The methodology may also depend on the accuracy and the distribution of the textual reports. Are the reports in the pretraining dataset automatically generated or written by cardiologists? 

The addition of “diagnostic data” introduces the possibility of learning data bias. What has been done to ensure generalization? 

It would be a good test for generalization to predict concepts not addressed in the textual reports e.g. if age and sex are not mentioned in the reports then a supervised task can be performed for these tasks, as these labels are readily available in most datasets.

How does the proposed approach differ from other hybrid approaches combining generative and contrastive methodologies e.g. [2]?

Is the text encoder frozen or fine-tuned?

The method may not be directly comparable to SSL methodologies as the pretraining is exposed to cardiac diagnostic texts enhancing the performance for related tasks. Performance for novel features, unrelated to the context of the reports (e.g. age and sex of the subject) may be impacted adversely. 

Please explain in further detail about where the N3S is applied as there are two cross-modal alignment losses. Is the cross-model alignment performed with single or multiple negative samples? If N3S avoids similar negative reports then the use of several positives and negatives in the batch can be incorporated in future work to avoid an expensive search for dissimilar negatives and looking explicitly for more distant reports is counter-intuitive to the goal of contrastive learning.

What is the difference between SigLEP and ETM losses? Don't they both serve the same purpose? Clearly explain whether the alignment is performed between projections of both modes, joint representation, or the inputs and how the positive and negative pairs are defined.

Does the application also provide the possibility of automatic ECG report generation?

The comparative approaches are not exposed to labels while textual reports may include the labels or similar concepts during pre-training.

Minor suggestions:

Page 1 line 23: Abstract: “achieving 15% and 2% increases.” Please mention the specific context of the percentages mentioned.

Page 2 lines (55-56): Introduction “While some recent efforts (Liu et al., 2024; Lalam et al., 2023; Li et al., 2024)”. Is the Li et al., 2024 repeated or have you missed the correct reference?

Page 2 lines (73-76): “Additionally, we introduce a nearest-neighbor negative sampling … contextually relevant and challenging.” How is the sampling “contextually relevant and challenging”?

Page 2 lines (77-85): No need to explain the test setup in the introduction. Please move to Section 4.1. 

Page 3:  Siglep is mentioned without explaining the acronym and the cited paper only describes “SigLIP”. Is it Siglep author’s implementation based on “SigLIP” or used in previous literature? If the latter, then please reference the correct source.

Page 4 line 165: Please explain the masking and reconstruction details e.g. is the masking only performed on particular leads similar to RLM in [3] and then the leads are reconstructed? Or is it also applied to segments of the same lead? 

Page 4 line (206-215): Is the MEM loss computed in the feature space, not the signal space? If the former then how does the reconstruction loss take into account the quality of the generated signals? There may be data leakage from the features that are input to the decoder as the network may learn trivial reconstruction. 

Page 6 line (273-274): “This makes the negative samples to be both challenging and distinct for effective contrastive learning”. The N3S technique for finding negative samples looks specifically for the most distinct reports. How does it make it more “challenging”? Also, the implementation is not very clear. Is the contrastive loss not considering the rest of the batch as negatives?

Page 6 line 313: “Our proposed model is developed based on the fairseq-signals framework in our work” The meaning of the sentence is not clear and the reference for the fairseq-signals framework is not included. If it means that is the author’s prior work then it might be a violation of the double-blind review process as the fairseq-signals framework is implemented on Git Hub.

Page 8 line 429: Have you tested removing the ETM loss?

Page 8 line (429-438): What is the supervised task?

Page 9 line (470): Have you considered using more than 8 eight transformer layers? How does the model size vary for the different architectures?

Tables: The particular metrics used are not specified in the table captions throughout the paper. Please mention the labeled dataset, the training configuration (linear probe or fine-tune), and the score in all table captions.

Tables: Are the comparisons in Tables 1 to 4 based on the author's implementation of other approaches? If so the performance for other works properly optimized for training hyperparameters for both the pre-training and supervised training? If not then do the cited references include the particular evaluation? 

Table 5: What is the metric being compared and why is the student seemingly better than a cardio resident? Please verify the scores.

Results: Some of the classes in the PTB-XL dataset have very few samples so for 1% and 10% random training samples there may not be sufficient positive samples in the train split. That can explain why SSL methodology loses performance significantly for labels other than superclasses. 

Figures: Please improve the figure captions and describe in more detail what is being shown.

References:

[1] Che Liu, Zhongwei Wan, Cheng Ouyang, Anand Shah, Wenjia Bai, and Rossella Arcucci. Zero-shot ecg classification with multimodal learning and test-time clinical knowledge enhancement. arXiv preprint arXiv:2403.06659, 2024. 

[2] Song, J., Jang, J. H., Lee, B. T., Hong, D., Kwon, J. M., & Jo, Y. Y. (2024). Foundation Models for Electrocardiograms. arXiv preprint arXiv:2407.07110.

[3] Jungwoo Oh, Hyunseung Chung, Joon-myoung Kwon, Dong-gyun Hong, and Edward Choi. Lead-agnostic self-supervised learning for local and global representations of electrocardiogram. In Conference on Health, Inference, and Learning, pp. 338–353. PMLR, 2022.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732100844450

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Revised Version

comment:
We thank all the reviewers for their comments and useful feedback. We have just uploaded a revised version of the paper to address comments, and we have also mentioned in the response to each reviewer. For convenience, we report the main differences to the previous version here:

 - Added experiments to demonstrate the effectiveness of ETM loss and MLM+MEM in sec. Appendix, Lines 828-852.
 - Improved introduction with motivations from the existing literature in Lines 54-76. 
 - Explained modifications to ECG encoder design (e.g. positional encoding, masking ratio) in Lines 162-172.
 - Improved explanation of our N3S technique in Lines 270-272.
 - Added suggested references, figure captions, and language adjustments.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732384866550

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Final Three Days

comment:
Dear Reviewers and Area Chairs,

We're entering the last three days of the discussion period. Four days ago, we addressed each reviewer’s comments and uploaded a revised version of our paper, which includes suggested adjustments and additional experiments. We are glad that Reviewer vCwP has updated their rating in light of our responses!

We would also greatly appreciate it if the other reviewers could have a look at our responses and consider adding their review if we have sufficiently addressed the remaining concerns. In particular, to Reviewer jws3, we humbly believe that our work deserves greater consideration than your current rating reflects. Should there still be unresolved questions, we would be happy to engage in further discussion.

Thank you for your time and consideration.
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper introduces C-MELT, a framework that pre-trains ECG and text data using a contrastive masked auto-encoder architecture. C-MELT combines the strengths of generative and contrastive learning through masked language and ECG modeling to reconstruct data. It also leverages contrastive loss functions and a nearest-neighbor negative sampling strategy to improve cross-modal alignment. Extensive experiments on five downstream datasets show good performance of C-MELT. The paper is generally well-written and clearly structured, and the experimental results are comprehensive and convincing. The pre-train data is MIMIC-IV-ECG and the downstream datasets include PhysioNet 2021, PTB-XL, CSN, CPSC2018, and CODE-test - all are publicly available standard databases. The code and pre-trained model are not included during the peer review (while the authors mentioned they would release them upon acceptance).

Reviewers appreciate the comprehensive experiments and detailed discussions. However, some major concerns from different perspectives still remain there. From an ML perspective, the contributions of the ECG-specific transformer, Flan-T5 text encoder, and GPT to enhance prompt quality are incremental. From an experimental perspective, some comparisons with MERL's best results should be provided. From a data perspective, the study has not generated new ECG data and/or new human annotations for resources in the field. From a clinical perspective, while C-MELT achieves higher performance, it doesn't provide additional functionalities to be implemented in clinical decision-making.