Title: GEM: Empowering MLLM for Grounded ECG Understanding with Time Series and Images
Venue: NeurIPS 2025
Accepted: yes
Keywords: Multimodal Large Language Model

ECG

Time Series

Medical AI
Abstract: While recent multimodal large language models (MLLMs) have advanced automated ECG interpretation, they still face two key limitations: (1) insufficient multimodal synergy between ECG time series and ECG images, and (2) limited explainability in linking diagnoses to granular waveform evidence. We introduce GEM, the first MLLM unifying ECG time series, 12-lead ECG images and text for grounded and clinician-aligned ECG interpretation. GEM enables feature-grounded analysis, evidence-driven reasoning, and a clinician-like diagnostic process through three core innovations: a dual-encoder framework extracting complementary time series and image features, cross-modal alignment for effective multimodal understanding, and knowledge-guided instruction data generation for generating high-granularity grounding data (ECG-Grounding) linking diagnoses to measurable parameters ($e.g.$, QRS/PR Intervals). Additionally, we propose the Grounded ECG Understanding task, a clinically motivated benchmark designed to comprehensively assess the MLLM's capability in grounded ECG understanding. Experimental results on both existing and our proposed benchmarks show GEM significantly improves predictive performance (CSN $7.4\%$ $\uparrow$), explainability ($22.7\%$ $\uparrow$), and grounding ($25.3\%$ $\uparrow$), making it a promising approach for real-world clinical applications. Codes, model, and data are available at https://github.com/lanxiang1017/GEM.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a multimodal LLM for ECG interpretation that supports feature-based analysis, evidence-based reasoning, and a clinician-like diagnostic workflow. Across several datasets, the method outperforms prior PULSE-based models and includes a cardiologist evaluation.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths\n\n1. Evaluation on multiple datasets shows the proposed approach consistently exceeds baseline methods.\n\n2. Qualitative assessment by cardiologists corroborates the model's effectiveness.\n\n3. The prompts used are provided, enhancing reproducibility.\n\n### Weaknesses\n\n1. The work is confined to ECG data; it lacks experiments or discussion on extending the method to other modalities or on the limitations imposed by this narrow focus. While this may suffice for a medical-domain venue, the broader audience of the present conference likely expects such analysis.\n\n2. GPT-4o is employed to create training data, yet its medical knowledge has not been evaluated here. Although GPT-4o scores highly on the USMLE—multiple-choice questions aimed at licensure—this study tackles tasks involving specialist reasoning, and the gap is non-trivial. An empirical assessment or a discussion in the limitations section is needed. It would also help to report how performance changes when data are generated and evaluated with models other than GPT-4o.\n\nQUESTIONS:\n1. Generalisability beyond ECG\n\n    Question. Could you either (i) add experiments on at least one additional medical modality (e.g., phonocardiograms, chest-x-ray images, or tabular vitals) or (ii) provide a principled analysis of why your architecture would—or would not—transfer to other signal types?\n\n    Actionable guidance. A small-scale ablation (≤1 additional dataset) or a synthetic test showing that the model can ingest non-ECG inputs would be sufficient.\n\n2. Dependence on GPT-4o-generated training data\n\n    Question. What is the quantitative effect of replacing GPT-4o with (a) a weaker open-source LLM or (b) clinician-curated annotations for the same prompts?\n\n     Actionable guidance. Please report performance when the synthetic corpus is produced by another model (e.g., MedAlpaca, or GPT-3.5) and, if feasible, when a small fraction is manually verified by cardiologists.\n\n3. Validation of GPT-4o's clinical reasoning depth\n\n    Question. Have you benchmarked GPT-4o on tasks closer to specialist reasoning (e.g., Board-style ECG interpretation questions with free-text answers) rather than multiple-choice USMLE items?\n\n    Actionable guidance. Reporting such a benchmark or summarising known literature—and clarifying how any shortcomings might affect label quality—would help contextualise the training pipeline's reliability.\n\nLIMITATIONS:\nThe authors should address the following critical limitation that appears to be missing from their current discussion:\nKnowledge inheritance and error propagation: If there are errors in GPT-4o's knowledge base, there is a possibility that these same errors may also be present in this model. This represents a significant limitation as any systematic biases, factual inaccuracies, or knowledge gaps from the source model could be perpetuated or even amplified in the current system. The authors should explicitly acknowledge this risk and discuss how they have attempted to identify, validate, or mitigate such inherited errors.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces GEM, a novel MLLM designed for grounded and interpretable ECG analysis. The authors address two key limitations in existing models: the insufficient synergy between ECG time series and image modalities, and the lack of explainability linking diagnoses to specific waveform evidence. To tackle these issues, GEM uniquely integrates a dual-encoder architecture for both time series and images, along with a cross-modal alignment mechanism to enable holistic understanding. A major contribution is the proposal of a knowledge-guided method for automatically generating ECG-Grounding, the first high-granularity instruction dataset that explicitly connects diagnoses to measurable physiological parameters. Furthermore, the paper introduces the Grounded ECG Understanding task, a new clinically-oriented benchmark designed to comprehensively evaluate the model's evidence-based reasoning capabilities.\n\nSTRENGTHS AND WEAKNESSES:\nS1. The proposed GEM model is the first to synergistically unify ECG time series, images, and text, establishing a novel multi-modal framework for more comprehensive ECG interpretation.\n\nS2. The creation of the ECG-Grounding dataset via a knowledge-guided generation process provides the high-granularity resource for training and evaluating evidence-based diagnostic models.\n\nS3. The introduction of the Grounded ECG Understanding task offers a new, clinically-oriented benchmark that enables a more thorough assessment of a model's explainability and grounding capabilities beyond traditional metrics.\n\nW1. The paper's parameter-efficient fine-tuning strategy, which freezes the encoders, may prevent the model from optimally adapting its feature extraction to the new grounding task. An exploration of full end-to-end fine-tuning is a notable omission.\n\nW2. The quality of the auto-generated ECG-Grounding dataset is not quantitatively validated by human experts, which is a critical step to ensure its reliability for training and evaluation.\n\nW3. The ablation study lacks an \"IMG-only\" baseline, making it difficult to fully assess the complementarity of the two modalities. Furthermore, the marginal performance gains from adding images in some tasks (Table 2) do not fully support the strong synergy claim.\n\nW4. The paper lacks a systematic analysis of failure cases, making it difficult to pinpoint whether errors stem from faulty feature extraction or flawed logical reasoning within the model.\n\nQUESTIONS:\nQ1: The Diagnosis Guider is a core component that relies on a predefined set of rules. Could you elaborate on its robustness? Specifically, how does this rule-based system handle rare or complex cardiac conditions that may not be explicitly covered by the existing rules, and what is its potential failure mode in such scenarios?\n\nQ2: The experimental results show a counter-intuitive finding where the GEM variant based on the general-domain LLaVA performs on par with, or even slightly better than, the variant based on the ECG-pretrained PULSE. Could you provide a possible explanation for this? Does this suggest that the pre-training strategy of PULSE might introduce a certain bias that is suboptimal for learning the new evidence-based grounding task you propose?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors introduce GEM, an MLLM for 12-lead ECG interpretation that fuses raw time series, rendered ECG images, and text prompts. A dual-encoder architecture aligns signal and image embeddings, which are then used to prompt a GPT-4o model to generate high-granularity instructional data for the LLM. Instruction-tuning on a 30K samples ECG-Grounding dataset lets GEM generate cardiologist-style reports that explicitly refer to waveform evidence. Experiments on a new Grounded-ECG benchmark and on standard classification tasks show reasonable gains in diagnosis accuracy and explanation quality over prior works.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. The paper is very well written, the task is well explained and motivated, and the GEM architecture is very well illustrated. The illustrations, in particular, make the paper easy to follow. \n\n2. The authors have done fairly extensive evaluation on eight different metrics, and the ablations justify the use of time series and image input. \n\n3. Strong reproducibility: The authors have made their code public, and the prompts and metrics are very well detailed in the appendix. The ECG-grounding dataset has also been made public.\n\n4. Strong performance gains over Pulse and GPT-4o for both in-domain and out-of-domain ECG-Understanding, and ECG-Bench abnormality detection. Evidence-linked explanations further improve transparency, and the qualitative examples show substantially better explainability than baselines.\n\nWeaknesses:\n\n1. Owing to the utilization of GPT-4o in the pipeline, I would an estimate API cost for training GEM. Similarly, inference time numbers are missing.\n\n2. The architecture, while novel, takes some inspiration from JoLT (Cai et al., 2023), which also does ECG interpretation, which isn't compared and contrasted with in the paper. \n\n3. Although this has been duly noted by the authors, GPT-4o interpretations might occasionally not align with those by experts. This is more critical in the healthcare domain, which GEM is focused on, compared to other domains.\n\nQUESTIONS:\n1. Did the authors run experiments substituting GPT-4o with an open-source alternative? If yes, what kind of a performance dip can one expect?\n\n2. Did the authors experiment with more LLM decoders? Did swapping those have a significant impact on performance, as observed in the case of JoLT (and BLIP-2)?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces GEM, designed to provide grounded, clinician-aligned interpretations of electrocardiograms (ECGs) by synergistically using ECG time series data, 12-lead ECG images, and text. The work makes several notable contributions, including a novel data generation pipeline, a new high-granularity dataset, and a clinically-oriented evaluation benchmark.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**:\n- The paper tackles a critical problem in medical AI: the lack of \"grounded understanding\" in automated ECG interpretation. Medical AI requires grounded reasoning to truly help clinician make informed decisions. \n- The proposed pipeline of creating grounded reasoning data could inspire research in scaling up medical AI. The release of the data could be of interest to the medical AI community.\n\n**Weaknesses**:\n\n- GPT-4o is not a board-certified cardiologist, and prone to hallucination and factual inaccuracies, especially in a high-stakes domain like medicine. The dataset, therefore, may contain subtle (or significant) errors, which are then propagated to the trained GEM model. The paper does not mention any quality control procedure on the created data.\n\n- Using GPT-4o to score the model's output is problematic, as \"LLM-as-a-judge\" approach can be both noisy and biased. The evaluation may reward outputs that are stylistically similar to GPT-4o's own generation style rather than those that are most clinically accurate. This introduces a potential for bias and reduces the objectivity of the results for the new benchmark.\n\n- Since the data generation pipeline largely depends on GPT-4o. This raises concerns about the cost, scalability, and reproducibility.\n\nQUESTIONS:\n1. Given that LLMs can produce factual inaccuracies, what specific quality control measures, if any, were implemented to validate the clinical accuracy of the generated interpretations? Were any samples reviewed by human experts?\n2. The proposed task is evaluated using GPT-4o, which introduces potential for self-reinforcement and stylistic bias. Can the authors comment on this limitation and the steps taken to ensure the evaluation is as objective as possible?\n3. The data generation method relies on GPT-4o. Could the authors provide an estimate of the computational cost and resources required to generate the 30,000-sample dataset?\n4. The cardiologist evaluation in Appendix A.4 highlights several cases where expert opinions differed from the interpretations generated by GEM or GPT-4o. How do the authors interpret these disagreements?\n5. Can the authors use candle plots to comprehensively reveal the stability of grounded ECG understanding for different models?\n\nLIMITATIONS:\n1. The primary limitation is the reliance on GPT-4o to generate the ground-truth labels for the ECG-Grounding dataset. As seen in the cardiologist evaluation, GPT-4o is not an infallible medical expert and can produce interpretations that are subtly or overtly incorrect.\n\n2. The use of GPT-4o as the evaluator for the Grounded ECG Understanding benchmark introduces potential biases. The metric may inadvertently reward stylistic similarity to GPT-4o's outputs over pure clinical accuracy. It could be beneficial if the authors could consider metrics that supports the trustworthiness of the LLM judgement.\n\n3. As with any clinical AI tool, there are potential risks. The primary risk is clinical error stemming from the model's recommendations, which could be compounded if the model was trained on flawed synthetic data. Over-reliance on the tool could also lead to \"automation bias,\" where clinicians may accept an incorrect AI suggestion. Therefore more analysis on reliability considerations could benefit the paper convincing the readers.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}