GEM: Empowering MLLM for Grounded ECG Understanding with Time Series and Images
Authors: Xiang Lan
Feng Wu
Kai He
Qinghao Zhao
Shenda Hong
Mengling Feng
Venue: NeurIPS 2025 poster ()
Keywords: Multimodal Large Language Model
ECG
Time Series
Medical AI
Primary area: applications

Abstract: While recent multimodal large language models (MLLMs) have advanced automated ECG interpretation, they still face two key limitations: (1) insufficient multimodal synergy between ECG time series and ECG images, and (2) limited explainability in linking diagnoses to granular waveform evidence. We introduce GEM, the first MLLM unifying ECG time series, 12-lead ECG images and text for grounded and clinician-aligned ECG interpretation. GEM enables feature-grounded analysis, evidence-driven reasoning, and a clinician-like diagnostic process through three core innovations: a dual-encoder framework extracting complementary time series and image features, cross-modal alignment for effective multimodal understanding, and knowledge-guided instruction data generation for generating high-granularity grounding data (ECG-Grounding) linking diagnoses to measurable parameters ($e.g.$, QRS/PR Intervals). Additionally, we propose the Grounded ECG Understanding task, a clinically motivated benchmark designed to comprehensively assess the MLLM's capability in grounded ECG understanding. Experimental results on both existing and our proposed benchmarks show GEM significantly improves predictive performance (CSN $7.4\%$ $\uparrow$), explainability ($22.7\%$ $\uparrow$), and grounding ($25.3\%$ $\uparrow$), making it a promising approach for real-world clinical applications. Codes, model, and data are available at https://github.com/lanxiang1017/GEM.

GEM: Empowering MLLM for Grounded ECG Understanding with Time Series and Images
Xiang Lan', Feng Wu', Kai He', Qinghao Zhao', Shenda Hong 204 Mengling Feng!
National University of Singapore "Peking University People's Hospital *Peking University *Correspondence to: hongshenda@pku. edu. cn, morninenus. edu.
Abstract
While recent multimodal large language models (MLLMs) have advanced auto- mated ECG interpretation, they still face key limitations: (1) insufficient multi- modal synergy between ECG time series and ECG images, and (2) limited explain- ability in linking diagnoses to granular waveform evidence. We introduce GEM, the first MLLM unifying ECG time series, 12-lead ECG images and text for grounded and clinician-alignedd ECG interpretation. GEM enables feature-groundee analysis, evidence-driven reasoning, and clinician-like diagnostic process through three core innovations: dual-encoder framework extracting complementary time series and image features, cross-modal alignment for effective multimodal understanding, and knowledge-guider instruction data generation for generating high-granularity grounding data LECG-Groundee
linking diagnoses to measurable parameters (e.g.. QRS/PR Intervals). Additionally, we propose the Grounded ECG Under- standing task, clinically motivated benchmark designed to comprehensively assess the MLLM's capability in grounded ECG understanding. Experimental results on both existing and our proposed benchmarks show GEM significantly improves predictive performance (CSN 7.4% 1), explainability (22.7% t), and grounding (25.3% t), making it promising approach for real-world clinical applications. Codes, model, and data are available at https:/giihub..oom/amiiingg
Introduction
Electrocardiography (ECG), cornerstone of cardiac diagnostics, captures the heart's electrical activity through body-surface electrodes, enabling non-invasive assessment cardiac physiology and pathology [Berkaya et al., 2018, Hannun et al., 2019]. Clinical ECG interpretation synergizes computational and clinical expertise: automated algorithms process raw signals detect fiducial points (e.g.. R-wave peaks) and generate diagnostic hypotheses, while clinicians validate these findings through 12-lead waveform analysis [Smulyan, 2019]. By contextualizing algorithmic outputs with -ppieit-specific factors, clinicians resolve ambiguities, detect subtle anomalies, and formulate diagnoses. This synergy between computational precision and expert judgment ensures reliable and holistic diagnoses in clinical practice.
Deep learning models have achieved promising results in cardiac anomalies detection [Hong et al., 2020, Zhu et al., 2021, Kiyasseh al., 2021, Lan et al., 2022, Yang et al., 2023, Zhao et al., 2024a, Li et al., 2024, Lan et al., 2024, McKeen et al., 2024] yet lack language capability and model explainability. While recent MILMs like PULSE [Liu et al., 2024b] have advanced language-based ECG interpretation via large-scale instruction tuning (e.g., ECG-Instruct's IM+ samples), they primarily focus static image inputs and predefined diagnostic tasks, leaving two critical gaps: Limited Modality Exploration. Current models typically process only one non-text modality, overlooking the benefits of synergistic approach that combining time series and image- based modeling. For example, time series models capture dynamic changes but may overlook spatial
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Oo Intersret
yzing the procided rC tracing, seaeral features ststrmalit
User
GEM
ayscercial tissuml
Evideecer warts
dit persistent
streres dites
2 Faadings: sinas mytts wtth oecastonsl premature
contractions
bxidenso: irreguaar ax intervals, arrticllr's the unterval
and corporiatary pause
following
517
3
Plndings: left ouss deviation
negattive coeples
It, ill, JS
4
Rind Rindings: iepcordial injury prior inforctiar
changes across
particularis in lateral precordial leat (1s, 16,
6
Pindings: right bastle branch black (1288) with left
anterior fiscicuaar block (LAPS)
1x128 es, tartioslarys
1. aul, and 46),
Conctusiass: overail, this atrormal ttD
exidence af prior ieocardial inforction, warranting cordia: celluction.
and
State-of-the-art MLLM:
irvca. ecs ba branch look (1888), and e left arter arteniar lassicalar block, suggestirg contaction assermalaties in undeterminet age, cntribbting to the everalt atrormulity of the Eec.
The electrecarttogee (EEG) reseate strax myths, intirating remall teart rate mytts, although
Sinus with (MCEs), Left axis deviation, 8958 with lett anterdor
secetereires, Nanornal COG.
Figure 1: GEM offers superior granularity in ECG interpretation compared to state-of-the-art models
and humtn-written reports.
patterns, while amage-based models detect global structures but may miss subtle temporal details. This limits their ability to replicate clinicians' holistic reasoning, which integrates both machine- measured temporal signals and waveform patterns from 12-lead plots. Insufficient Explainability and Grounding. Most existing models designed for ECG provides limited explainability, failing to explicitly connect diagnoses granular waveform evidence and provide insight into their diagnostic reasoning. A trustworthy ECG model should not only predict cardiac conditions but also explicitly highlight which ECG features led to those conclusions. Such grounded explanations enhance transparency and make model outputs more reliable and actionable for clinical decision- decision-making.
In this work, we introduce GEM, multimodal large language model designed for grounded ECG understanding by integrating time series, image, and text data. As conversational cardiology AI assistant, GEM differs from other MLLMs through three key features. First, it provides feature- grounded analysis, ensuring that its findings are explicitly tied to detailed ECG features. Second, offers evidence-driven diagnosis, where its conclusions are supported by clear and logical reasoning directly linked to ECG findings. Lastly, GEM simulates a realistic interpretation process, mimicking how clinician analyzes ECGs and arrive at diagnosis.
Achieving these capabilities entails challenges in two key dimensions. On the modeling side, is crucial to effectively integrate information from different modalities to support accurate ECG understanding. On the data side, there is currently no available instruction data designed for training LLMs on gighhgranllartty ECG interpretation.
We tackle these challenges through three novel approaches. Multimodal Encoding allows GEM to extract and integrate complementary features from raw ECG time series and their transformed images. It employs dual-encoder architecture, with each encoder specialized in its respective modality using established models from the time series and vision domains. This design leverages the unique strengths of both modalities. Cross-modal Alignment Learning facilitates the interpretation of multimodal ECG data by the LLM. Time series representations are first projected to image representations dimensionality, followed by shared projector that transforms both into language- like embeddings that are comprehensible to LLM. These aligned embeddings are then fused with textual instruction embeddings, enabling effective multimodal understanding through next-token prediction training. Knowledge-guizn high-granularrtyy instruction data annotated with heartbeat-level physiological features, without the need for manual annotation. This methodology integrates a grounding feature extractor, which derives precise physiological features from ECG time series, and a gardiology-specingg diagnosis guider, which processes these features into structured prompts to more effectively leverage GPT-4o's
Instruction Data Generation supports the construction of

---

latent medical knowledge for generating clinically detailed and feature-grounden grounded ECG instruction data. Ultimately, GEM delivers significantly more detailed and informative interpretations than both human-written reports and leading MI.MMs, as shown in Figure 1.
The main contributions of this work three-folds:
1. 1.First Unified Multimodal ECG Model. We present GEM, the first multimodal framework to synergistically integrate raw ECG time seriesa, 12-lead ECG plots, and textual instructions, leveraging their complementary strengths to advance grounded ECG understanding.
2. First granularry ECG Grounding Dataset. We propose novel knowledge-gune guided instruction data generation method, resulting SCG-Grounding, dataset comprising 30,000 instruction pairs annotated with heartheat-level physiological features. This is the first nigh-granulaare ECG grounding dataset, enabling evidence-basse -ased diagnosis and improving the trustworthiness of medical AI.
3. Clinically Oriented Diagnostic System. We introduce the Grounded ECG Understanding task, a clinically motivated benchmark designed to comprehensively assess a model's ECG interpretation capability. Experimental results demonstrate that GEM not only excels in predictive performance but also significantly enhances explainability and grounding, making it more applicable for real-world clinical settings while fostering greater trust among medical professionals.
2
Related Work
2.1 Multimodal Large Language Models
Large Language Models (LLMs), such as GPTs [Achiam et al., 2023], LLaMA [Touvron et al., 2023], have made significant advancements in artificial intelligence. Despite their superior performance on numerous natural language processing tasks, LLMs are inherently limited to the text modality, making them "blind" to other modalities such as images, audio and video. To mitigate this constraint, Multimodal Large Language Models have been recently developed to extend the ability of LLMs in comprehending multiple modalities [Liu al., 2024a, Zhang et al., 2024, 2025, Huang et al.. 2025]. By integrating LLMs with various data sources, MLLMs enable the handling of diverse information beyond text. For example, LLaVA [Liu al., 2024a] enables LLMs to comprehend visual inputs by adopting a learnable projector to map image features into the word embedding space. Video-LLaMA [Zhang et al., 2023] further enhances LLMs by enabling video perception and understanding. Qwen-Audin [Chu et al., 2023] introduces an audio- audio-language model capable of processing various audio types, including human speech, natural sounds, and music. Medical data, by nature, are inherently multimodal, encompassing diverse formats such as images, physiological time series, and textual reports. These modalities collectively form the foundation for clinical decision- making, driving the advancement of AI in medical applications [Li et al., 2023, Moor et al., 2023, Radhakrishnan et al., 2023, Liu et al., 2023, Hong and Hong, 2023, Zhu et al., 2024, Ren et al., 2024, Sellergren et al., 2025, Yang et al., 2025, Jin et al., 2025, He et al., 2025]. Different from these works, our focus is on empowering MILMMs with the grounded ECG understanding capability.
2.2 Language-based ECG Analysis
Language-based ECG diagnosis and interpretation is still in its early stages of development. Only a few recent studies have explored LLM-based approaches for ECG analysis. For instance, Yu et al. [2023] proposes zero-shot retrievallaugmmented diagnosis technique, embedding domain knowledge from textbooks and research papers into a vector database to improve zero-shot diagnostic accuracy. Cai et al. [2023] proposes JoLT, framework that jointly models ECG time series and text using Querying Transformer to align their representations. Liu et al. [2024b] introduces PULSE, an LLM-based framework designed to enhance ECG image understanding for diagnosis and report generation. PULSE synthesizes realistic ECG images from raw ECG signals, enabling better utilization of image-based LLaVA models. Zhao al. [2024b] develops ECG-CoCa, an ECG encoder trained on ECG-text pairs, alongside ECG-Chat, a modified LLa VA model capable of processing ECG time series. Chan et al. [2024] proposes an analytical framework integrating time series data with L.Mss, combining physiological signal analysis with contextual textual information. Wan et al. [2024] designs an instruction- -unnnng framework for automated ECG report generation, converting ECG-text pairs into chatbot-style instructions and fine-tuning the LLM's 's linear layers.
3

---

Language Respanses
Large Language Model
<Image> embeds
<TS> embeds
Knowledge Guided In Instruction Duta Generation (test)
Query: "Iavyyet the provider ECG date <Ioagee =TS-, idontly lay Jonnress and sbsorenttier eoch lent and goverate a cindictt diogsnnts that supponed by the otsenved evidence Wles anaiszin the ECG iaye, curefully snabes cach load, Load E: Emssine tie Q85 onpinaiic and duration along with ST repeest and Twise worpholog...
15-Image-Text Projector
TS-Image Projector
Cross-modde cerees-odaat...o Alignment Learning
Target Answer: "The DCG descourrreee rlnss rkyihis will occasional prosunire vaurirular connections (PPC)) exidewced by Imegolor Hll mocroall, partiullrty the slavee interaa of 255 su and rie conpenatory paue folloning it 537
ECG Encoder
ECG Encoder (TS)
GPT-4a
(Image)
Diagnosis Guider
Multimodal Encoding
104, ST_tom': upsiace...
transformed image
raw signal
Grounding Feature Extractor
Figure 2: GEM's Architecture. Multimodal Encoding: Separate encoders process ECG time series and images to generate modality-specifie representations, enabling a holistic analysis of ECG data. Cross-modal Alignment Learning: Time series and image representations are first aligned and then mapped to a textual space using a shared projector, ensuring coherent understanding for the LLM. Knowledge-guiin Instruction Data Generation: Physiological features extracted from all 12 leads are sequenced and structured using a diagnosis guider, which prompts GPT-4o with domain-spectie instructions generate high-granularity instructional data.
GEM distinguishes itself from existing language-based ECG models in two key aspects. First, in model architecture, unlike existing methods that limit analysis isolated modalities (either ECG signals or transformed ECG images) with text, GEM introduces a unified architecture that integrates both time series data and 12-lead images. This approach mirrors a clinician's 's natural workflow, where dynamic signal trends and spatial waveform patterns are jointly analyzed for more comprehensive interpretation. Second, ECG understanding, GEM establishes new paradigm for -rrienneddriven diagnosis. While current models often lack grounded understanding, GEM enables heartbeat- interpretability by directly linking each diagnostic conclusion to quantifiable physiological evidence, enhancing explainability and clinical reliability. By combining the two paradigm-shifting innovations, GEM addresses fundamental limitations existing models and advances language-based ECG analysis, setting new standard for conversational Al-assisted cardiac diagnostics.
3 Method
3.1
Overview
GEM's training primarily relies on three key components: the multimodal encoding, the cross-modal aligmment learning, and the knowledge-guided instruction data generation. Figure provides an overview of the GEM model. We will elaborate on each components in the following sections.
3.2 Multimodal Encoding
We represent the ECG time series Tts E CCLL, where C' is the number of leads in multi-lead ECG and L is the length of the signal. The transformed ECG image derived from the time series denoted Ding I #x xxx33 Wx3 where H and W are the height and width of the image, respectively. For ECG time series encoder (ft,,, we adopt the pre-trained ECG-CoCC model [Zhao et al., 2024b], which has been extensively trained on large number of ECG-Text pairs with contrastive learning to effectively capture the intricate patterns within the ECG time series:
ets R xd,,
xd, (1ee
(I)

---

where is the number of time series feature patches and d, is the dimension of time series features. For ECG image encoder ging (f (mig)) we utilize the pre-trained CLIP encoder from LLaVA [Liu et al., 2024a]. This model is adept at understanding and processing visual information, making it suitable for extracting features from the ECG images:
esng
I Pmm Xdm Esmyl Timgl [8mmg) )
(2)
where fmm is the number of image feature patches and dm is the dimension of image features. These two encoders enable the separate extraction of features from time series and images. This dual- encoder approach allows us to harness the distinct advantages of each data type, enhancing the overall interpretative power of our model.
3.3 Cross-modal Alignment Learning
Considering that the time series and image encoders trained independently, their generated representations often show inconsistencies within the representation space. Mcanwhile, to ensure that the LLM can interpret ECG time series and images effectivelyy, is essential that these multimodal inputs are rendered as comprchensible as textual data. Therefore, aligning these diverse modality inputs within a unified representation space becomes crucial.
In our approach, the ECG time series representation is first mapping to the same dimensionality as the ECG image representation. This is accomplished using multi- -lyeer perceptron (MLP) projector:
IR xdm LL (e:all0 (..))
C
Subsequently, we employ additional projector to map both the time series and image representations into a consistent textual space:
h1s E Ru. = MM (9ss)
(4)
h Bing E xd= -MLP(eiggl0al
(5)
where de is the dimension of the text embeddings. This step is for ensuring that the features extracted from both modalities are not only aligned dimensionally but also interpretable in a LLM-friennlly format.
Once we have obtained the features from both the time series and image modalities, we integrate these with the embeddings of the textual query Zgg:
Concatenate h hang: Embeded (r.)))
6
The integration is crucial for creating a cohesive representation that encapsulates the full spectrum of information from the multimodal inputs.
3.4 Knowledge-guided Instruction Data Generation
Instruction data forms the foundation of multimodal training, directly shaping how the MLLM generates responses for given queries. This is because the language response 0L (x) optimized to match the target answer for each multimodal input (æts: Zing: zgy). Here, we introduce two key mechanisms to guarantee that y aligns with paauure-grondder analysis, evidence-driven diagnosis, and realistic interpretation process, thereby empowering GEM with a grounded understanding of ECG data.
Grounding Feature Extractor. To enable the feature-grounder analysis, we propose to further excavate more detailed information from raw ECG time series. We begin by extracting universal elements, including waveforms, amplitudes of fiducial points, and intervals, from each heartbeat in each lead. These elements are then structured into a time-ordered sequence for further analysis. For instance, an ECG time series with ten visible heartbeats allows us to construct a QRS duration sequence as [QRS,, QRS2. ..., RS1], where each element precisely represents the QRS duration for specific heartbeat. These sequences provide fine grade description of the physiological features of the ECG, enabling the model to assess heart conditions at the level of individual heartbeats. In our implementation, we incorporate 14 feature sequences for each of the 12 ECG leads, covering: Heart Rate, RR Interval 1, RR Interval 2, Amplitude P Duration, PR Interval, QRS Amplitude, QRS Duration, T Amplitude, T Duration, ST Duration, ST Form, QT Interval, and QTc Interval.
5

---

ECG-Instruct data (PULSE)
ECG-Grounding data (Ours)
Query: Maat the axis of QRS complex this ECG?
Query: Interpret provided isage, identify tty key features and abnormalities in each lead, and generate clinical diagnosis that 15 supported by the observed exidence.
A. Norm'l axis B, Right axis deviation C. Left axis deviation D. Indeteriinte axis
Asswer: ECS demonstrates sinus rhythe with occasional prenature ventricular contractions (PVCS), as evidenced by irregular BR intervals, particularly ly the shorter interval at 255 as and the compessatory passe following it at 517 ms. The left axts ceviation is suggested by predosingg insst ly negative 0RS complex is Leads II, III, and aVF, indicating an abnorssl leftward ortentation
shows which characterized by predoninsrtly negative This indicates that electrical
Target Answer:
ion,
positive complex lead
deviated the left, fron the the is correct option C. Left axis deviation
renleeeaar depal ap izat inn right burdle branch block (R888) with anterior rascicular is
duery: Investigate the ECC results and compile thorough report.
with rsR' particutarly in Leads VS-V6, characteristic waves in and avi with persistent SI-T abnornalities suppests a previous lateral infarct of undetermined age, whien is consistant with infarcted syocardial tissue. Furthermore, ST-T changes across na naltiple leads, par aerticllarly in the lateral precordial lends (V5, ve, avL,, indicate myocardial injury prior infarction, Overall, combination conduction disturbances, PVEs, clinical prior ayecardial
tiiple leads and V6),
aVL, R006. The presence of seall
deep
Target Answer: electrocardiograr (ECS) reveals rhytte,
rhytte, although interrupted premature ventricular contractions (PVCs), The devistion,
exhibits left axis
bundle branch block (8008), anterior fascicular block, suggesting conduction intornalities in the heart's is electrical systes. adcitionally, the ECC shows evidence of a laterat infaret, or ayocard ial. infarction, undeteraised age, contributing to the overall abnormality of the ECG.
left
an abnorral with a
possibly addit
evaluation.
Liedtations
Advantages
1-Occasionally Lyy inaccurate Ired textl when LUT lacks
1. F.osture-Grounded Anslysis: The and paalssss is based on specific
reference information,
ECG features and explicitly reference these features as evidence.
2.lack detai led explasse explanations that explicitly ly connect
2. S.vvidena--Driven Diagnosis: the diagrosis is supported by clear,
diagnoses grans lar wavefore ev evineeee.
lngical reasoning tied to the FCG findings.
Wate: tratersal
lafe auis Deviationn
3.ttmlatee Realistic Integretation Process: The interpretation diagnesis.
085 is positivv in lood
negontive lead 1t, alF
analyse
Figure 3: Comparison of ECG-Instruct and our ECG-Grounding.
This comprehensive set of sequences captures the temporal evolution of key physiological features, enabling granular analysis of cardiac activity. The feature extraction process is formulated as:
f Fe)
where ze is a dictionary in which the keys represent feature names and the values correspond to their respective feature values. FeatureDB (x+.) represents the function that extracts structured physiological features from the ECG time series Ets- Note that there is no trainable parameters in FeatureDB.
Diagnosis Guider. With the feature sequences fys extracted, the challenge lies in generating high- granularity y without relying on costly human-expert annotation. To address this, we design diagnosis guider that constructs a prompt Tp to effectively guide GPT-4o in generating accurate and clinically grounded responses y for each sample:
Pp = DiagnosisGuider(zzs.)
(8)
The diagnosis guider provides cardiology-specinng instructions for detailed analysis of each aspects of ECG data (e.g., instruct GPT-4o to assess the P wave amplitude and duration in Lead II to evaluate atrial enlargement) and incorporates guidance reflecting real-world clinical diagnostic processes. As result, each sample receives unique H, tailored to its fts ensuring precise activation of GPT--4o's latent medical knowledge for accurate and personalized analysis. See Appendix A. for more details of the diagnosis guider.
ECG-Grounding Data. Using the knowledge-guided instruction data generation method, we employ GPT-4o to curate 30,000 fine-grained instruction-response pair (æx y) from the MIMIV-IV-ECG
[Gow et al., 2023] database:
y = GPTT440xxp))
A comparison illustrating the advantages of our instruction data is shown in Figure 3. Instruction data from PULSE, primarily derived from the original report, occasionally provides incorrect explanations due to hallucinations, which can arise when LLM has limited reference information. In contrast, our CCG-GGrunding provides more accurate, holistic, and evidence-driven interpretations with diagnoses grounded in measurable ECG features. Training with these fine-grained instructional data, GEM significantly enhances its explainability and grounding capabilities, making it a promising approach for clinical applications and fostering greater trust among medical professionals.

---

3.5 Training
Unlike conventional multi-step training pipelines for most MLLMs, which first train linear projector using brief image captions for gross-modal alignment and subsequently fine-tune the LLM with instruction data, we adopt an one-step training approach for GEM. In this approach, we freeze Bt and fL.Ma- This one-step process enhances the consistency in training multiple modalities with limited data and improves the efficiency of the training phase.
and Ding for feature extraction, while jointly training Ma, ОM,
The training objective is formulated as minimizing the negative logliiklidood of the target answer y and the LLM response OLLM (x), given the embedding of multimodal input x:
Tly; (LLLM())) = -
E log P(ys x, OLLM), :=1
(10)
where N represents the number of tokens in and Mj is the j-th token in y. We train two foundational LLMs for the GEM framework. The first, LLaVA, is the unmodified version of the original LLaVA- 7B model, which has not undergone training on any ECG data. The second, referred to as PULSE, represents tftt--ffthhe-ar LLMs specifically trained on millions of ECG images. For both models, we implement supervised fine-tuning (SFT) for single epoch. We use 8 A100 GPUs for the training.
Experiments
4.1 Training Data
The training of GEM involves two datasets: the ECG-Instruct data from PULSE [Liu et al., 2024b], which includes 1,156,1 1,66,110 10 conversations, and our generated ECG-Grounding data, comprising 30,000 conversations. The ECG-Grounding data is sampled from MIMIC-IV-ECG [Gow et al., 2023], selecting only samples that have not been used in training models such as PULSE. We use the ECG-image-kit (Shivashankara et al., 2024] for the generation of ECG images from the original ECG signal, and use the Fe [Hong al., 2017, 2019] extract ECG features. To facilitate further research and benefit the community, we have publicly released BCG-Grounding data.
4.2 Evaluation Tasks and Metrics
Grounded ECG Understanding. To comprehensively evaluate whether the model achieves clinically grounded ECG interpretation capabilities comparable cardiologists, introduce the Grounded ECG Understanding task. This task is developed based on cardiologist guidelines and evaluates the MLLM's ability to identify detailed diagnostic clues in ECG analysis, requiring to provide specific details and relevant clinical knowledge to support its interpretation.
We utilize GPT-4o to score the responses of the MLLM using a predefined set of metrics that measures the accuracy and comprchensiveness of the details provided. Specifically, these metrics include: DiagnosisAccuracy, Analysis SonedlssisCompletennesss AnalysisRelevance, LeadAssessmentCCveene LeadAssessmenttccuuraa
Evidence BasedReasoningg and ClinicalDiagnos- ticFidelity. The definitions and interpretations of each metric provided in Appendix A.2, and further details on the scoring methodology for each criterion are outlined in Appendix A.3.
The evaluation is conducted on the test sets of two datasets: MIMIC-IV-EGG (2,381 samples) and (TB-Xer et al., 2020] (2,041 samples). For the grounded ECG understanding task, MIMIC- IV-ECG serves as the in-domain dataset, while PTB-XL. represents an out-domain dataset, with test samples drawn from a different data distribution.
ECG-Bench In addition, we use the ECG-Bench [Liu et al., 2024b] to assess our model's capability in cardiac abnormality detection and report generation. ECG-Bench is comprehensive benchmark for evaluating MLLMs on ECG understanding. It incorporates several datasets including the PTB-XL dataset, CPSC2018 dataset [Liu al., 2018]. G12EC dataset [Alday al., 2020], CODE-15% dataset [Ribeiro et al., 2021], and CSN dataset [Zheng et al., 2020].
contains two main tasks: abnormality detection and report generation. For the abnormality detection task, uses AUC, F1, and Hamming Loss (HL) as metrics for multi-label datasets, and accuracy for others. In the report generation task, it employs GPT-4o to evaluate the reports based on their accuracy rhythms, waveform descriptions, and diagnoses, with maximum score of 100.
7

---

Table 1: Grounded ECG Understanding results MIMIC-IV-ECS and PTB-XL.
Metric
Diagnosis Accuracy
Analysis Completeness
Analysis Relevance
Lead Assessment Coverage
Lead Assessment Accuracy
Feature Grounding
ECG
Evidence Based Reasoning
Diagnostic Fidelity
Clinical
MIMIC-IV-ECG (in-domain)
PULSE
81.14
2.37
2.39
7.11
2.95
50.18
52.40
51.63
GEM (Ours)
SFT LLaVA
87.24
4.41
5.01
71.07
46.44
75.48
75.09
75.28
SFT PULSE
86.49
4.43
69.80
45.33
74.95
74.70
74.87
PTB-XL (out-domain)
PULSE
59.24
2.20
2.06
11.20
6.27
52.52
55.48
53.85
GEM (Ours)
SFT LLaVA
73.53
4.19
2.96
79.54
49.01
74.48
74.61
73.84
SET PULSE
73.59
4.19
3,00
78.86
47.96
74.97
75.41
74.24
Table 2: ECG-Bench abnormality detection results.
Datasets
PTB-XL Super
CODE-15%
CPSC 2018
CSN
G12EC
Metric
AUC
F1
HL
AUC
F1
HL
AUC
F1
HL
ACC
ACC
Random
50.3
33.2
50.1
48.8
15.0
32.1
51.2
28.8
11.6
12.1
GPT-4o
55.6
28.3
26.2
59.9
24.9
15.7
50.9
10.6
57.5
49.2
PULSE
82.4
74.8
11.0
90.7
85.4
5.0
76.9
85.2
78.2
GEM (Ours)
SFT LLaVA
81.8
73.6
11.6
90.5
84.8
5.1
74.1
52.0
9.0
92.6
81.8
SFT PULSE
83.4
75.8
11.0
91.5
86.4
4.7
79.1
61.1
8.1
86.2
80.5
Ablations
TS only TS+MGG
81.2
72.5
11.9
90.8
84.9
5.0
76.3
54.0
91.6
81.4
82.7
74.8
11.1
91.3
86.3
4.6
74.4
51.5
90.1
81.1
4.3 Results
Table summarizes the performance of GEM the Grounded ECG Understanding task across both in-domain (MIMIC-IV-ECH) and out-domain (PTB-XL) datasets. The proposed GEM models consistently outperform the state-of-the-art PUL.SE model across all evaluation metries. For diagnosis accuracy, GEM achieves 87.24% on MIMIC-IV-ECG, surpassing PULSE (81.14%) ()) by over 6%. On the out-domain dataset, GEM maintains robust generalization with accuracy of 73.59%, again outperforming PULSE (59.24%) by a notable margin of 14.35%. This remarkable performance on the out-domain dataset highlights GEM's strong reasoning capability, enabling generalize effectively and make accurate diagnoses across diverse data distributions. In analysis completeness and relevance, GEM demonstrates substantial gains. On MIMIC-IV-EGG, improves completeness from 2.37 to 4.43 and relevance from 2.39 to above 5.01. Similar trends are observed on PTB-XL, where GEM doubles the PULSE's completeness and improves relevance by over 45%. This substantial performance gap indicates that GEM not only identifies and interprets greater number of critical ECG components (with an average two-fold increase feature coverage) but also maintains stronger clinical relevance by effectively connecting these observations to the diagnostic reasoning process. Lead assessment coverage and accuracy also improve significantly. On the in-domain dataset, GEM increases coverage from 7.11% % to over 71% 11 and accuracy from 2.95% to above 46%. Improvements on the out-domain dataset rising from 1 11.00 to over 79% for coverage and from 6.27% over 49% for accuracy. These gains verify GEM's ability to perform structured and precise evaluation across multiple ECG leads, an essential skill for cardiologist-
reasoning.
Regarding ECG feature grounding, GEM achieves scores around 75 on both datasets, indicating that large portion of diagnostic conclusions are explicitly linked to measurable ECG parameters. This significant advancement over PULSE, which achieves scores only around 50. Furthermore, GEM outperforms PUL.SE in wridence-based reasoning and clinical diagnostic fidelity, with both metrics exceeding 74 across datasets. These results show GEM's ability to construct clinically coherent justifications and align with structured diagnostic processes, key for real-world clinical integration.
In the ECG-Bench task, Table 2 offers detailed examination of the models' performance on cardiac abnormality detection across a variety datasets.

---

The results clearly show that our GEM (SFT PULSE) model consistently outper- forms other models. Furthermore, the GEM (SFT ILLVA) version, which has not been previously trained on ECG data, still manages to achieve comparable, and in some cases superior (e.g., 7.4% in CSN and 3.6% in GI2EC), performance across most datasets, despite being trained for only single epoch. These results highlight the robustness and efficacy of the GEM framework, demonstrating its capacity to deliver substantial performance gains even with minimal domain-specific training. Ablation studies further highlight the critical role of multimodal inputs in achieving efficient training and superior classification performance. We assess two model variants: TS-only, which is trained exclusively time series and image data. Notably, the TS+IMG model surpasses PULSE on PTB-XL Super and CODE-15% datasets, despite being trained for only one epoch. In Table 3, GEM showcases exceptional report generation capabilities, achieving a substantial 5.8% improvement over PULSE in the PTB-XL Report while maintaining comparable performance in ECG-QA. These results further evident GEM's 's capability in delivering holistic, accurate, and clinically-alinned ECG interpretations.
Table 3: ECG-Bench report generation and QA results.
Datasets
PTB-XL Report
ECG-QA
Metric
Report Score
Accuracy
Random
0
16.2
GPT-4o PULSE
50.2
35.2
61.3
73.8
GEM (Ours)
SFT LLaVA
65.0
71.0
SFT PULSE
67.1
73.6
ECG time series, and TS+IMG, which incorporates both
Collectively, GEM demonstrates superior performance in both Grounded ECG Understanding and ECG Bench tasks, establishing its effectiveness in ECG interpretation across multiple dimensions.
4.4 Cardiologist Evaluation
We conduct a structured cardiologist evaluation covering three sources of outputs: GPT-4o generated training data, Deepseek-R1 generated training data, and GEM generated interpretations.
In total, 400 ECG-Grounding data (200 from GPT-4o and 200 from Deepseek-RI Guo et al. [2025]) and 200 GEM's interpretation were independently reviewed by eight board-certified cardiologists, using seven predefined clinical criteria designed assess both real-world reliability and usefulness. This unified evaluation protocol allows us to (1) verify the quality of GPT-40 generated training data, (2) test the effectiveness of open-soure substitutes, and (3) confirm the clinical utility of the GEM model. See Appendix A.4 for detailed scoring criteria.
We report the evaluation results in Table 4 and 5 below. The expert evaluation shows that GPT-4o consistently achieves high scores across both reliability and clinical usefulness, with particularly strong performance in analytical completeness and reasoning quality. These results demonstrate that, when using our sinoweede-gguide instruction data generation, GPT-4o is capable of generating high-quality ECG interpretations that are both clinically reliable and practically valuable.
Deepseek-RR R1 is also capable of generating clinically acceptable, high-quality ECG interpretations with our methods. This demonstrates that our method is adaptable to alternative LLM backbones and remains applicable in settings without commercial API access.
The expert evaluation results also demonstrate that GEM consistently achieves high scores across both reliability and usefulness dimensions, with most metrics rated above 4 out of 5. These findings indicate that GEM is capable of producing clinically meaningful and accurate interpretations that align well with cardiologists' expectations. High scores in reliability-relate metrics reflect the factual correctness and clinical grounding of its outputs, while strong performance in usefulness metrics suggests that GEM is not only technically sound but also practically helpful in supporting diagnostic decision-making, Collectively, these results support GEM's potential trustworthy assistant for real-world cardiology applications.
In Appendix A.5, we showcase six representative cases involving complex cardiac conditions, in which GEM's interpretations exceeded expert expectations. In these cases, cardiologists highlight two types of findings: (1) those they have not noticed during their own ECG examination, which exceed their expectations for a cardiology Al assistant in real-world settings, and (2) those where they hold differing opinions.

---

Table 4: Evaluation of reliability metrics by cardiologists (Mean and STD).
Analytical Relevance
Analytical Accuracy
Analytical
Model
Completeness
GPT-4o
4.7/5 (0.66)
4.6/5 (0.82)
4.7/5 (0.65)
Deepseek-RI
4.8/5 (0.57)
4.7/5 (0.78)
4.9/5 (0.42)
GEM
4.6/5 (0.60)
4.4/5 (0.80)
4.6/5 (0.57)
Table 5: Evaluation of usefulness metries by cardiologists (Mean and STD).
Reasoning Quality
Findings Novelty
Clinical Value
Overall
Model
Satisfaction
GPT-4o
4.7/5 (0.67)
4.4/5 (1.18)
4.7/5 (0.73)
4.5/5 (0.87)
Deepseek-R1
4.8/5 (0.62)
4.5/5 (0.91)
4.6/5 (0.82)
4.7/5 (0.77)
GEM
4.6/5 (0.64)
3.9/5 (1.25)
4.3/5 (0.89)
4.4/5 (0.82)
Overall, GEM demonstrated its capability to generate clinically insightful findings, often surpassing expert expectations by identifying details that cardiologists have not noticed, suggesting its potential for real-world clinical applications. It is noteworthy that cardiologists also highlights certain cases where their interpretations diverged from GEM or GPT-4o. Although our knowledge-guided instruc- tion data generation approach avoids costly expert annotations while producing high-quality target answers, GPT-40 still occasionally generates target answers that may be misaligned with cardiologist interpretations. For example, in Figure 5, the GPT-4o suggest no evidence ischemia or infarction, while cardiologist suspects there are ischemia in the precordial leads. These deviations highlight opportunities for future refinement of model reasoning with human feedback to better align with cardiologist level clinical judgment.
4.5 Failure Case Analysis
We also conduct an analysis of failure cases informed by expert feedback from the human evaluation. The main errors fall into two categories. The first is incorrect diagnosis, often caused by limitations in the representation stage. Subtle morphological patterns such as ST-segment changes or P-wave abnormalities were occasionally missed by the encoders, likely due to feature representation defi- ciencies or insufficient training data coverage of diverse ECG variations. The second is overstating the severity of findings. GEM occasionally exaggerated the severity of certain cardiac conditions, which cardiologists noted could lead to unnecessary patient concern. This tendency may result from the absence of patient context, as the model interprets ECGs without access to clinical history that physicians would normally consider. These findings help clarify the current limitations and inform future directions to enhance model safety, reliability, and clinical alignment.
Conclusion
In this work, we present GEM, the first MLLM for grounded ECG interpretation, integrating ECG time series, 12-lead ECG images, and textual instructions. GEM achieves feature-grounder analysis, evidence-driven diagnosis, and clinician-style diagnostic workflows through three core technical innovations: multimodal encoding, cross-modal alignment learning, and showledge-guided instruc- tion data generation. The multimodal encoding and cross-modal alignment learning allow the LLM to simultaneously process ECG time series and image representations, effectively leveraging the instruction
complementary strengths of both modalities for interpretation. The knowledge-guide data generation addresses the lack of high- granularity instruction data for ECG understanding. dataset comprises 30,000 fine- ame-grained instruction pairs annotated with heartbeat-leve physiological features, which establishes the first high-resolution resource for grounded ECG understanding. We also introduce the Grounded ECG Understanding task, a clinically motivated benchmark that comprehensively assesses models' grounded ECG understanding through multi-dimensional metrics. Together, these contributions establish a solid foundation for future works in conversational diagnostic Al for ECG interpretation.
The developed
10

---

Acknowledgment
This research is supported by A*STAR, CISCO Systems (USA) Pte. Ltd., and the National University of Singapore under its Cisco-NUS Accelerated Digital Economy Corporate Laboratory (Award 121001 E0002), and the Al for Public Health Program Saw Swee Hock School of Public Health, National University of Singapore. Shenda Hong is supported by CCF-Tencent Rhino-Bird Open Research Fund (CCF-Tencent RAGR20250109), and CCF-Zhipu Large Model Innovation Fund (CCT-Zhipu202414))
References
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 2023.
Erick A Perez Alday, Annie Gu, Amit Shah, Chad Robichaux, An-Kwok Ian Wong, Chengyu Liu, Feifci Liu, Ali Bahrami Rad, Andoni Elola, Salman Seyedi, et al. Classification of 12-lead ecgs: the physionet/computing in cardiology challenge 2020. Physiological measurement, 41(12): 124003, 2020.
Selcan Kaplan Berkaya, Alper Kursat Uysal, Efnan Sora Gunal, Semih Ergin, Serkan Gunal, and M Bilginer Gulmezoglu. A survey ecg analysis. Biomedical Signal Processing and Control, 43: 116-235,220 2018.
Yifu Cai, Mononito Goswami, Arjun Choudhry, Arvind Srinivasan, and Artur Dubrawski. JoLT: Jointly learned representations of language and time-series. In Deep Generative Models for Health Workshop NeurIPS 2023, 2023. URL hwwph:ttt/// //pprr/veee///eeee/////ee net/ forummiddUVEE AMMB Ju.
Nimeesha Chan, Felix Parker, William Bennett, Tianyi Wu, Mung Yao Jia, James Fackler, and Kimia Ghobadi. Leveraging llms for multimodal medical time series analysis. In Machine Learning for Healthcare Conference. PMLR, 2024.
Yunfei Chu, Jin Xu, Xiaohuan Zhou, Qian Yang, Shiliang Zhang, Zhijie Yan, Chang Zhou, and Jingren Zhou Qwen-audio: Advancing universal audio understanding via unified large-scale audio-language models. arXiv preprint arXiv:2311.070 2023.
Brian Gow, Tom Pollard, Larry A Nathanson, Alistair Johnson, Benjamin Moody, Chrystinne Fernandes, Nathaniel Greenbaum, Jonathan W Waks, Parastou Eslami, Tanner Carbonati, al. Mimic-iv-ecg: Diagnostic electrocardiogram matched subset. Type: dataset, 6:13-14, 2023.
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Peiyi Wang, Qihao Zhu, Runxin Xu, Ruoyu Zhang, Shirong Ma, Xiao Bi, al. Deepseek-rl incentivizes reasoning in I1ms through reinforce- ment learning. Nature, 645(808 1018116:333388 2025.
Awni Y Hannun, Pranav Rajpurkar, Masoumeh Haghpanahi, Geoffrey Tison, Codie Bourn, Mintu P Turakkhia, and Andrew Y Ng. Cardiologist-leva arrhythmia detection and classification in ambulatory clectrocardiograms using deep neural network. Nature medicine, 25(1):65-69) 2019.
Kai He, Rui Mao, Qika Lin, Yucheng Ruan, Xiang Lan, Mengling Feng, and Erik Cambria. A survey of large language models for healthcare: from data, technology, and applications to accountability and ethics. Information Fusion, 118:102963, 2025. ISSN 1566-2535. doi: https:
2025. 102963.. URL https: //ww:://////w////////////:p/ /www sciencedirect. com/ aom/scient science/ arricil/pp11/11665555500033
Haoyang Hong and Shenda Hong. simplenomo: a python package of making nomograms for visualizable calculation of logistic regression models. Health Data Science, 3:0023, 2023.
Shenda Hong, Meng Wu, Yuxi Zhou, Qingyun Wang, Junyuan Shang, Hongyan Li, and Junqing Xie. ENCASE: an ensemble classifier for ECG classification using expert features and deep neural networks. In CinC, 2017. doi: 10.24899Cin.2201.1778822 URI. https: //doi org/10. 22489/ CAAa9//Cin... 2017 178-245-

---

Shenda Hong, Yuxi Zhou, Meng Wu, Junyuan Shang, Qingyun Wang, Hongyan Li, and Junqing Xie. Combining deep neural networks and engineered features for cardiac arrhythmia detection from ECG recordings. Physiological Measurement, 40(5):054009 Jun 2019. doi: 10. (0.1088/1311-5577 ab15a2. URL. htt.ttttt///o..... org. 1068//213111 -11116666666991966622222
Shenda Hong, Yuxi Zhou, Junyuan Shang, Cao Xiao, and Jimeng Sun. Opportunities and challenges of deep learning methods for electrocardiogram data: A systematic review. Computers Biology and Medicine, page 103801, 2020.
Jiaxing Huang, Jingyi Zhang, Kai Jiang, Han Qiu, Xiaoqin Zhang, Ling Shao, Shijian Lu, and Dacheng Tao. Visual instruction tuning towards general-purpose multimodal large language model: survey. International Journal of Computer Vision, pages 1-39, 2025.
Jiarui Jin, Haoyu Wang. Xiang Lan, Jun Li, Gaofeng Cheng, Hongyan Li, and Shenda Hong. Uniecg: Understanding and generating ecg in one unified model. arXiv preprint urXiv:2509.18558 2025.
Dani Kiyasseh, Tingting Zhu, and David A Clifton. Cloes:
Contrastive learning of cardiac signals across space. time, and patients. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 5606-5615, PMLR, 18-24 Jul 2021. URL https: http://epocceen.....//
mlr. www.ppeeseeeee /vt/s///v1//// wwl.vv19//iii http///aasssseaaaaaaaaaaaaaaaaaa html.
Xiang Lan, Dianwen Ng. Shenda Hong, and Mengling Feng, Intra-inter subject self-supervised learning for multivariate cardiac signals. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 4532-4540, 2022.
Xiang Lan, Hanshu Yan, Shenda Hong, and Mengling Feng. Towards enhancing time series con- trastive learning: dynamic bad pair mining approach. In The Twelfth International Conference on Learning Representations, 2024. URL https: http::///ppeeeeeeeeee
Chunyuan Li, Cliff Wong, Sheng Zhang, Naoto Usuyama, Haotian Liu, Jianwei Yang, Tristan Naumann, Hoifung Poon, and Jianfeng Gao. Llava-med: Training a large assistant for biomedicine in one day. arXiv preprint arXiv:2306. 00890, 2023.
Jun Li, Che Liu, Sibo Cheng, Rossella Arcueci, and Shenda Hong. Frozen language model helps eog zero-shot learning. In Ipek Oguz, Jack Noble, Xiaoxiao Li, Martin Styner, Christian Baumgartner, Mirabela Rusu, Tobias Heinmann, Despina Kontos, Bennett Landman, and Benoit Dawant, editors, Medical Imaging with Deep Learning, volume 227 of Proceedings of Machine Learning Research, pages 402-415. PMLR, 10-12 Jul 2024. URL https: / http://pooceeeeeeeeeeee/e/c mlr. wwwppree//// v227
1i24a.htm.
Feifei Liu, Chengyu Liu, Lina Zhao, Xiangyu Zhang, Xiaoling Wu, Xiaoyan Xu, Yulin Liu, Caiyun Ma, Shoushui Wei, Zhiqiang He, al. An open access database for evaluating the algorithms of electrocardiogram rhythm and morphology abnormality detection. Journal of Medical Imaging and Health Informatics, 8(7):166--1375 2018.
Fenglin Liu, Tingting Zhu, Xian Wu, Bang Yang, Chenyu You, Chenyang Wang. Lei Lu, Zhangdai- hong Liu, Yefeng Zheng, Xu Sun, et al. A medical multimodal large language model for future pandemics. NPJ Digital Medicine, 6(1):226, 2023.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36, 2024a.
Ruoqi Liu, Yuelin Bai, Xiang Yue, and Ping Zhang. Teach multimodal 1lms to comprehend electro- cardiographic images. arXiv preprint arXiv:2410.19000 2024b.
Kaden McKeen, Laura Oliva, Sameer Masood, Augustin Toma, Barry Rubin, and Bo Wang. Eeg-fm: An open electrocardiogram foundation model. arXiv preprint arXiv:2408.05177 2024.
Michael Moor, Oishi Banerjee, Zahra Shakeri Hossein Abad, Harlan M Krumholz, Jure Leskovec, Eric J Topol, and Pranav Rajpurkar. Foundation models for generalist medical artificial intelligence. Nature, 616(7956):25--22 2023.
12

---

Adityanarayanan Radhakrishnan, Sam F Friedman, Shaan Khurshid, Kenney Ng. Puneet Batra, Steven A Lubitz, Anthony Philippakiss and Caroline Uhler. Cross-modal autoencoder framework learns holistic representations of cardiovascular state. Nature Communications, 14(1):2436 2023.
Wenhui Ren, Zheng Liu, Yanqiu Wu, Zhilong Zhang, Shenda Hong, Huixin Liu, and Missing Data in Electronic health Records (MINDER) Group. Moving beyond medical statistics: A systematic review on missing data handling in electronic health records. Health Data Science, 4:0176, 2024.
Antônio H Ribeiro, GM Paixao, Emilly M Lima, M Horta Ribeiro, Marcelo M Pinto Filho, Paulo R Gomes, Derick M Oliveira, Wagner Meira Jr, Thômas B Schon, and Antonio Luiz P Ribeiro. Code-15%: A large scale annotated dataset of 12-lead ecgs. Zenodo, Jun, 9, 2021.
Andrew Sellergren, Sahar Kazemzadeh, Tiam Jaroensri, Atilla Kiraly, Madeleine Traverse, Timo Kohlberger, Shawn Xu. Fayaz Jamil, Cían Hughes, Charles Lau, al. Medgemma technical report. arXiv preprint arXiv:2507.05201, 2025.
Kshama Kodthalu Shivashankara, Afagh Mehri Shervedani, Gari D Clifford, Matthew Reyna, Reza Sameni, et al. Eeg-image-kit: synthetic image generation toolbox to facilitate deep learning-based electrocardiogram digitization. Physiological Measurement, 45(3):055019, 2024.
Harold Smulyan. The computerized ecg: Friend and foe. The American Journal of Medicine, 132 URL
(2):153-160, 2019. ISSN 0002-9343, doi: http:////n///ecccccc//////////// https://www sci sciencedirect. wtt.ce/irret..com/cceeeeeeeeeeee
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint
13971,
Patrick Wagner, Nils Strodthoff, Ralf-Dieter Bousseljot, Dieter Kreiseler, Fatima Lunze, Wojeiech Samek, and Tobias Schaeffter. Ptb-xl, a large publicly available electrocardiography dataset. Scientific data, 7(1):1-15, 2020.
Zhongwei Wan, Che Liu, Xin Wang, Chaofan Tao, Hui Shen, Zhenwu Peng, Jie Fu, Rossella Arcucci, Huaxiu Yao, and Mi Zhang. Electrocardiogram instruction tuning for report generation. arXiv preprint ar arXiv:2403 04945, 2024.
Chaoqi Yang, M Westover, and Jimeng Sun. Biot: Biosignal transformer for cross-data learning in the wild. Advances Neural Information Processing Systems, 36:78240-78260. 2023.
Kai Yang, Massimo Hong. Jiahuan Zhang, Yizhen Luo, Suyuan Zhao, Ou Zhang, Xiaomao Yu, Jiawen Zhou, Liuqing Yang, Ping Zhang, et al. Beg-Im: Understanding electrocardiogram with a large language model. Health Data Science, 5:0221, 2025.
Han Yu, Peikun Guo, and Akane Sano. Zero-shot ecg diagnosis with large language models and sttreval-augmeen generation. In Machine Learning for Health (ML4H), pages 650-663. PMLR, 2023.
Hang Zhang, Xin Li, and Lidong Bing. Video-llama: An instruction-tuned audio-visual language model for video understanding. arXiv preprint arXiv:23306.0 02858, 2023.
Jingyi Zhang, Jiaxing Huang, Sheng Jin, and Shijian Lu. Vision-languag models for vision tasks: survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, 46(8):5625-5644 2024. doi: /LL09/TPAMM.2224.333666
Jingyi Zhang, Jiaxing Huang, Huanjin Yao, Shunyu Liu, Xikun Zhang, Shijian Lu, and Dacheng Tao. R1-vI: Learning to reason with multimodal large language models via step-wise group relative policy optimization. arXiv preprint arXiv:2503. 12937, 2025.
Qinghao Zhao, Shijia Geng, Boya Wang, Yutong Sun, Wenchang Nie, Baochen Bai, Chao Yu, Feng Zhang, Gongzheng Tang, Deyun Zhang, et al. Deep learning in heart sound analysis: from techniques clinical applications. Health Data Science, 4:0182, 2024a.
Yubao Zhao, Tian Zhang, Xu Wang, Puyu Han, Tong Chen, Linlin Huang, Youzhu Jin, and Jiaju Kang. Ecg-chat: large eog-language model for cardiac disease diagnosis. arXiv preprint arXiv:2408. 08849, 2024b.
13

---

Jianwei Zheng, Jianming Zhang, Sidy Danioko, Hai Yao, Hangyuan Guo, and Cyril Rakovski. A 12-lead electrocardiogram database for arrhythmia research covering more than 10,000 patients. Scientific data, 7(1):48, 2020.
Lingxuan Zhu, Weiming Mou, Keren Wu, Yancheng Lai, Anqi Lin, Tao Yang. Jian Zhang, and Peng Luo. Multimodal chatgpt-4v for electrocardiogram interpretation: Promise and limitations. Journal of Medical Internet Research, 26xe54607, 2024.
Zhaowei Zhu, Xiang Lan, Tingting Zhao, Yangming Guo, Pipin Kojodjojo, Zhuoyang Xu, Zhuo Liu, Siqi Liu, Han Wang, Xingzhi Sun, et al. Identification of 27 abnormalities from multi-lead eeg signals: An ensembled se_resnt framework with sign loss function. Physiological Measurement, 42(6))065008,, 2021.
14

---

GEM: Empowering MLLM for Grounded ECG Understanding with Time Series and Images Appendix
A.1 Diagnosss Guider Prompt
16
A.2Grounded ECG Understanding Metries
17
A.3GPT-40 Evaluation Prompt
18
A.4Scoring Criteria for Cardiologist Evaluation
20
A.5Case Studies by Cardiologist
22
A.6 L.orrrderr Impacts and Limitations
28
15

---

A.1 Diagnosis Guider Prompt
Your task: Interpret provided ECG image, identify key features and abnormalities in each lead, and generate clinical diagnosis that is supported by the observed evidence.
## Key objectives:
1. Simulate Realistic Diagnostic Process: The interpretation should reflect how a doctor would analyze an ECG, ask clarifying questions, and arrive at diagmosis.
2. Grounded ECG Understanding: The analysis should be based on specific
ECG features and explicitly reference these features evidence.
3. Evidence- Based Reasoning: The diagnosis should be supported by clear,
logical reasoning tied to the ECG findings.
## Guidelines for the ECG analysis:
1. Data:
ECG image: an image that display the 12-lead ECG tracings. Make the task centered on the ECG image, assuming direct ECG image analysis.
Machine measurements: A time-ordered list of ECG features computed for each heartbeat every lead. Each entry in the list corresponds features calculated for a single heartbeat.
2. Act cardiologist and medical knowledge to analyze the provided ECG image step-by-step:
Initial Analysis: Analyze the provided ECG image identify key features such as rhythm, intervals, and any apparent abnormalities.
Detailed Reasoning: Explain your thought process step-by-step, referencing specific ECG features (e.g., "The ST segment is elevated in leads V1-V4, which suggests anterior myocardial infarction").
Svidence-Base Diagnosis: Propose diagnosis or differential diagnoses, justifying your conclusions with explicit ECG data.
3. When analyzing the ECG image, carefully analyze each lead: Lead I: Examine the QRS amplitude and duration, along with ST segment and T wave morphology. Abnormalities may indicate lateral wall issues such left ventricular hypertrophy, bundle branch block, or lateral sichemia/infaattion
amplitude and duration assess right left atrial enlargement; the PR interval can reveal conduction delays. ST and T wave changes here suggest inferior wall ischemia or infarction. Leads III and aVF: Primarily reflect inferior wall status. Abnormal Q waves, along with segment and T wave changes, point toward inferior infarction or ischemia.
Lead II: Look the P
Lead aVL: Focuses on high lateral region; QRS, ST, and T wave abnormalities here suggest high lateral ischemia or infarction.
Lead aVR: ST elevation may indicate left main or multivessel disease, and wave inversion can be associated with ventricular arrhythmia.
Lead V1: An increased R wave, a characteristic rSR pattern, and ST-T changes help identify right ventricular hypertrophy, right bundle branch block, or ischemia.
Leads V2-V4: Assessing the anterior anteroseptal regions. The presence of Q waves, along with ST segment and wave deviations, suggests anterior wall infarction or ischemia.
Leads V5-V6: Focus on the lateral wall, where similar QRS, ST, and T wave changes indicate lateral ischemia infarction.
4. When analyzing the machine measurements, you should aware that: a. If any abnormalities appear in the computed measurements that are not
16

---

mentioned in the report, you must strictly follow and trust the report. b. Evaluate and interpret the machine measurements as if you had computed them yourself. In your analysis, refer to these values as your own computed measurements rather than using phrases like "machine measurements provided".
## Guidelines for the response generation:
1. Synthesize your findings to deduce a likely diagnosis or set of diagnoses. Clearly explain how the evidence supports your conclusion.
2. Ensure your diagnosis is comprehensive and strictly based on the report.
Do not include diagnosis that not mentioned in the report.
3. Make sure your diagnosis are grounded in the given ECG image and machine measurements, and you should explicitly reference (e.g., specify lead and the position of the abnormal heartbeat).
4. Strictly follow the output format and requirements specified in your task instructions.
5. The given report only served the ground truth for you to analyze the ECG image. The generated text must not show that you are aware of the
existence of the report.
6. Never make explanations.
## ECG Report: [[report]]
## ECG Machine Measurements:
batachine_mmasuremene
## Generation rule The generated text must not show that you are aware of the existence of the report. Do not include phrases like "Based the report", or "Given the ECG report". The primary objective is to analyze the ECG and identify evidence that supports the results. The analysis should focus solely on the ECG itself, never analyze the report.
## Present your work in this format:
*Rssponse: [Comprehensive response following the task's guidelines, strictly based the report. Using complete paragraph with more natural expression. Do not use a list format. Limit your responses within 300 words.]
A.2 Grounded ECG Understanding Metrics
DiagnosisAccuracy evaluates whether the generated diagnosis is correct, specific, and supported by ECG findings. Results are expressed as a percentage, indicating the average accuracy across identified key diagnoses.
PhulysisCompletenes checks if all key ECG components (e.g., rhythm, intervals, waveforms, and lead-specific findings) are discussed. Results are provided in absolute terms, indicating the average number of correctly addressed key ECG features for each sample.
AnalysisRelevane assesses whether each explanation directly supports the diagnosis, with results showing on average how many points support the diagnosis with clear ECG evidence for each sample.
LeadAssesmmenCCveergge evaluates how many of the 12 ECG leads are analyzed. Results indicate the average percentage of leads analyzed per sample, providing insight into the comprehensiveness of the ECG assessment.
LeadAssessen seaddsssesmentccccraay verifies the accuracy of described lead findings (e.g., QRS, ST, T waves, amplitude, intervals, ST segments) against the ground truth interpretation. The result reflects the average percentage of accurately identified findings across the 12 ECG leads.
17

---

SCGFeatureGrounding determines if the interpretation references actual ECG features (e.g., QRS amplitude, PR interval) instead of generic terms. Results are scaled from 0 100.
BeveBeenness evaluates whether the diagnosis follows logical, evidence-supported steps. Results range from 0 100.
CiinicaDiagnoxxticideiii assesses if the model mimics how clinician interprets ECG data, consid- ering all relevant factors. Results are scaled from 0 to 100.
A.3 GPT-4o Evaluation Prompt
Your task: Evaluate the alignment and quality of a generated ECG interpretation by comparing it to ground truth clinician's interpretation.
## Evaluation Criteria:
1. DiagnosisAccuracy: Evaluates whether the generated diagnosis is correct, specific, and supported by ECG findings.
Scoring
+2 per diagnosis: Each correctly identified key diagnosis with supporting
ECG features.
+1 per diagnosis:
Each mostly correct diagnosis but lacking key supporting
details.
+0 per diagnosis: Each incorrect or vague diagnosis not supported by ECG features.
2. AmalysisCompletenes:: Checks if all key ECG components (rhythm, intervals, waveforms, and lead-specific findings) discussed. Scoring
+1 per feature: For each correctly addressed key ECG feature (e.g., rhythm, PR interval, QRS duration, ST segment, T wave morphology). per missing feature: For each key feature omitted or inaccurately described.
3. AnalysisRelevance: Assesses whether each provided explanation directly supports the diagnosis. Scoring
+2 per feature or per lead: Each point that strongly supports the diagnosis with clear ECG evidence.
+1 per feature or per lead: Some points are relevant but not fully justified.
+0: Includes unrelated or misleading explanations.
4. LeedAssessmentCCverag:: Evaluates how many of the 12 ECG leads are analyzed.
Scoring
+1 per lead: For each lead adequately assessed.
+0 per missing lead: For each lead omitted or inaccurately described.
5. LeachsesmentAcuraay: Checks if the described lead findings (e.g., QRS, ST, waves, amplitude, intervals, ST segments) match standard ECG interpretation.
Scoring
+2 per lead: Findings closely match expected values.
+1 per lead: Findings are somewhat accurate but have minor
inconsistencies.
+0 per lead: Findings contradict ECG norms.
6. EOGFeatureCromding: Determines the interpretation references
actual ECG features (e.g., QRS amplitude, PR interval) instead of generic terms.
Scoring (0-100)
18

---

100: ECG findings are comprehensively cited, linked to diagnoses, and cover all relevant ECG features.
80: ECG findings are explicitly cited and linked to diagnoses.
50: Some ECG references exist but are incomplete.
0: Lacks specific waveform references.
7. Ev.denceBasedRessoning Evaluates whether the diagnosis follows logical, evidence-supporter steps. Scoring (0-100)
100: Findings logically progress to diagnosis with thorough and clear justifications covering necessary steps.
80: Findings logically progress to diagnosis with clear justifications.
50: Some reasoning exists but lacks complete step-by-step analysis.
0: Reasoning is unclear not derived from ECG findings.
8. Cinicalbiaguosticiieeell.
Assesses if the model mimics how clinician interprets an ECG, considering all relevant factors. Scoring (0-100)
100: The analysis follous a structured clinical approach and considers all relevant clinical factors.
80: The analysis follows a structured clinical approach.
50: Some clinical reasoning is present incomplete.
0: The approach lacks structured clinical reasoning.
NOTE: Each score must be calculated based strict criteria to objective evaluation.
## Generated ECG Interpretation:
(fmodel_generateel)
## Ground Truth Clinician's Interpretation: [ggoonntrrthh]
## Response Format:
Provide your evaluation strictly in the JSON format below. For any criterion with multiple elements (e.g., multiple diagnoses or leads),
list each one separate "Score": X, "Explanation": "Y" entry. Use
a single entry for criteria with aggregate scores (e.g., 0-100 scores).
{ "DiagnosisAccuraccy":
"Score": 2, "Explanation": "Sinus tachycardia
correctly identified supported by short interval.", "Score": 1, "Explanation": "Left ventricular hypertrophy is mostly correct but lacks QRS amplitude detail."
AmnalysissComllettnesss: [ "Score": 1, "Explanation": "PR interval
is correctly described.", "Score": 1, "Explanation": "QRS duration assessed. "Score": 0, "Explanation": "ST segment not addressed." J,
MmnalysieRelevvnce" [ "Score": 2, "Explanation": "QRS prolongation supports diagnosis of bundle branch block."
"LeadsseessennCCvvraag"""" [ "Score"; 1, "Explanation": "Lead I assessed.", "Score": 1, "Explanation"; "Lead assessed.", "Score":
"Explanation": "Leads V4-V6 omitted."
"Leadkesesaeenttcuracy"" [ "Score": 2, "Explanation": "Findings in Lead II match standard interpretation." "Score": 1, "Explanation": "Lead III slightly misinterpretedd but largely accurate." ".
"ICGFaatureGrouddigg. "Score": 80, "Explanation"; "Most findings cite ECG features like QRS and T wave, but some vague." 1,
"Ev.denccBaeddeesoinn""" "Score": 100, "Explanation": "Diagnosis is built on step-wise reasoning with reference to all major findings." ],
19

---

"Oiinicaliiagposticccelll"""""" [ "Score": 80, "Explanation": "Analysis
mimics clinician structure but misses minor climical context."
A.4
Scoring Criteria for Cardiologist Evaluation
Table 6: Reliability metrics.
Criterion
Description and Scale
Analytical Relevance
Does the model's analysis closely support the diagnosis and provide corre- sponding ECG evidence?
(1-5)
5 Every analysis point is highly relevant to the diagnosis, with clear sup-
porting evidence.
4 Most analyses strongly relevant, with minor insufficiencies.
Some analyses are relevant, but there is clear irrelevant content.
Most analyses weakly relevant.
The analysis is unrelated to the diagnosis.
Analytical Accuracy (1-5)
Completely accurate. Are there any medical factual errors in the model's output?
4 Mostly accurate.
Some errors.
Obvious errors.
Severe errors.
Analytical Completeness (1-5)
Does the model comprehensively discuss key ECG components relevant to the diagnosis, including rhythm, intervals, and waveforms?
- All relevant ECG features (rhythm, PR, QRS, ST, T waves, intervals,
etc.) are accurately discussed.
4
Most key ECG features are covered, with minor omissions.
Only some features are covered, with significant gaps.
Only few ECG features are mentioned.
ECG components are largely missing, with severe omissions.
20

---

Table 7: Usefulness metrics.
Criterion
Description and Scale
Reasoning Quality (1-5)
Does the model provide clear, evidence-basee reasoning process similar that of clinician, logically deriving the diagnosis from ECG features?
Clear and coherent reasoning structure, explaining each step from ECG to diagnosis causally.
Overall reasonable reasoning, but some steps lack detail.
Partial reasoning present, but incomplete or logically weak.
Disjointed reasoning with major gaps.
1 No logical reasoning, only stack of conclusions.
Findings Novelty (1-5)
Does the model provide insights findings not noticed by the clinician?
Important new diagnoses or findings.
Novel and somewhat insightful content.
Some new findings, but of limited value.
Conventional content, not particularly insightful. 1 No new information.
Clinical Value (1-5)
Does the model output help in clinical decision decision-making? making?
- Direct and significant support for clinical judgment; content clear and reliable.
Most content is helpful and practically useful.
Somewhat informative, but basic or unclear.
Partially suggestive, with limited decision support.
No value for clinical judgment; informative.
Overall Satisfaction
Subjective rating of the overall output quality.
(1-5)
Very satisfied.
Satisfied.
Neutral.
- Dissatisfied.
Very dissatisfied.
21

---

A.5 Case Studies by Cardiologist
36
96
GEM Interpretation: Upon enalyzing the provided ECG image and computed messurements, several key features and ebnormalities lies are evident. The heart rate is elevated 121 bpm, indicating rapid ventriculer response. The rhythm eppears irregular, consistent with strial fibrillation. es seen by the absence of distinct P and the presence of irregulrr BR intervals. the precordial leads, particularly V1 through V4, there are could suggest conditions such pericardial elfusion obesity QRS complexes in these leads are notably diminished, which aligns with the low valtage finding Turthermore, the presence premature ventricular contractions (PVCs) suggested by the imreglarr RR intervals and the occasional to are
low QRS vollages
abcence praceding waves. The STT changes ooserved the anterolateral leads, particularly nonspecific but could indicate ischemic changes or other cardlac conditions The OTc intervals are prolonged several leads, which may suggest nik for arrhythmas) The combination of these findings, including the repid ventricular response, irregular rhythm,
nonspedilic ST-T changes, supports clagnosis strial fibrillation with raoid ventricular response and PVCs. low CRS voltages
GRS voltages,
precorcial
leads add another layer of complesity. potentially indicating underlying cardiac concitions. The ECG is clessified abnormal due these findings, which warrent further clinical correlation and possibly additional a diagnostic testing to determine the underlying caure these abnormelities.
Target Answer (GPT-40): Upon examining the provided ECG, we identily several critical festures and abnormal ties. The first notable obsarvation the presence of atrial fibrillaiion charactenzed by iregular rhythm and lack distinet waves the ECG tracings. This is supported by the irreularr RR intervals, is, which vary signilicntly between heartbeats. The heart rate is elevated at 132 bpm, indicative of repid ventricular response, Additionally, the ECG reveals frequent premeture ventricular contractions, or multifocal PVCs, with verying QRS complex merphologies, suggesting aberrant ventricular conduction ectopy. Lead and precordial leeds demonstrete low QRS volages, which can correlate with various conditions, inclucing myncardial infarction myopathics. terms STT changes, the anteriar leads (V1 V4) may suggest possible entenior myocardial abnormal rios. Moranver, infarior and lateral leads, partioularly lead avi, display nonspacific. ST.T changes, which could indicate ischemia other pathological processes. comprehensive review of the QT intervel reveals prolonged QTc intervals particularly
possible ventricular
as evidenced
amplitude
morphology
precordial leads, reflecting altered repolanzation processes. Furthermore no distinct high. lsteral, or inferior leed pattern suggests definitive infarction, but the chenges are significant enough warrent further clinical investigation and correlation with clinical symptoms and history. Conclsivvlyy the FOG findings strongly support diagnosis atrial fibrillation with rapid ventricular resporse, accompanied by frequent multifocal PVCs or eberrant ventricular conduction. The anterior region's ST- changes and precordial low QRS voltages emphasize the need for further assessment, possibly indicating previous ongoing ischemic event. However, additional clinical context and investigatins are essential corroborate these findings and formulate complete treatment strategy.
Ground truth Report: Atrial fibvillation with rapid ventricular resporse with frequent multifocal PVCs aberrant ventricular conduction; Possible enterior infarct; Inforior/lateral ST. changes nonspocific, low QRS vokages precordial leads: Abnormal ECG.
resporse, aocompanied by premature ventricular contractions (PVCs) or aberrant ventricular conduction, ind cating imegglar heart rhythm. Additionally, y. the ECG suggests possible anterior myocarcial infarction, although the age infarct cannot determined. inferior and lateral ST-T changes norspecific, the vollages are low in the precordial leads, contributing to an overall abnorml ECG pettern.
PULSE Interprotation: The elecrocerdiogran (ECG) reveals fibrillaionn rapid
Figure 4: Cardiologist Evaluations. Blue: Findings exceeding expert expectations. Yellow: Findings with differing expert opinions.
22

---

Whn Ve
Min
ICmmueiV
GEM Interpretation: Upon enalyzing the provided ECG image and computed measurements, several key features and abnormalities are evident. The rhythm irregular, with heart of 102 bpm, which is consistent with atrial fibrillation. This is supported by the absence of distinct P waves across multiple loads, in the measurements. The QRS complexes are generally narrow, indicating supraventicular ongin, which aligns with atrial fibr llatinn Lead the ORS amplitude and duration are within normal limits, but the irregular rhythm is evident. Lead il shows a similar pattern with alssent waves and irregular intervals, further supporting atrial librillation. Leads and aVF, which reflect the inferior wall, do not show significant Q waves or ST segment changes that would suggest acute ischemia or infarction. Lead aVi. focusing on the high lateral region, also does not exhibit significant abnormalitics in the ORS, ST, or wave morphology. Lead aVR, often used to assess global ischemic changes, wave inversion. which might like left main coronary artery disease In the precordia al leads V1 through V6, the QRS complexes are narrow, depressions that would suggest acute myocardial infarction. Overall, the ECG findings are consistent with atrial fibrillaiion, characterized by the absence of distinct P waves irregular RR intervals. The lack significant segment changes pathological waves across the leads suggests that there is no acute ischemic event present This analysis supports the diagnosis of atrial fibrillation the primary abnormality, with evidence scute myccardial infarction other significant ischemic ischemicchanges..
such as Lead and Lead II, and the presence of irregular RR intervals,
does not show significant
indicated more severe conditions
and there no significant ST segment elevations
Target Answer (GPT-40): Upon examining the provided ECG image, several key features indicate the presence of atrial fibrillation (AF). The imagularly irregular rhythm, as evidenced by varying RR intervals, supports this observation. Notably, the RR intervals range widely, which halkmark of AF due to incorsistent atrial impulses reaching the ventricles. Additionally, the absence of distinct waves leads with tracitionally observable P waves is another pivotal indicator; this suggests a loss of organized atrial electrical activity, consistent with with a presentation of AF with controlled rate. No consistent pattern in the PR intervel is observed, further corroborrttngg diagnosis of AF, as atrial activity chaotic and not leading to predictable conduction pattern through the AV nade. Analyzing leads such as load II, where P wave characteristies are more observable under normal conditions, inconsistency of PR intervals signify typical conduction during AF episodes, contributing to its erratic rhythm. In QRS and wave deviations across various leads
epproximates 107 bpm, indicative
moderate ventricular rete, aligning
see minimal absent wave formations. The variability and
of additional abnormalinies, not form pathological pattern significant enough to suggest additional diagnoses such as ischemia of infarction, The ST segment and QTc intervals within normal limits given context of AF, This ECG analysis supports diagnosis of atrial fibrillation, underpinning the clinical picture with evidenced imogular BR intervals and absent waves, interpretation remairs focused the features aligning with AF without assurning further conditions not supported by the ECG findings Ground-truth Report: Atrial fibrillation.
noted, they
PULSE Interpretation: The elactrocand (ECG) reveals atrial fibrillaiion, characterized imegular heart thythm with no discemiibe waves, indicating lack of coordinated atrial contraction. Additionally, the ECG shows repolarization abnormality, suggesting potential issue with the heart's recovery phese after contraction. The probable rate-related abnormalityy implies that the repolarization abnormality may be related to the atrial fibrillation, which causing imregllar heart rate.
Figure 5: Cardiologist Evaluations. Blue: Findings exceeding expert expectations. Yellow: Findings with differing expert opinions.
23

---

25mpvs
r GEM Interpretation:
Upon analyzing the provided ECG image and computed measurements, several key features and abnormalities evicient. The rhythm sinus, as indlcctedd by the consistent
morphology across leads and a regular heart rate of 75 hpm- In Lead the ORS complex shows a prolonged duratton, which is consistent with right bundle branch block (RBBB). This is further supported by the characteristic raft
pettern Lead V1, hallmark of RBBB. The QRS duration Lead iis notably prolonged, ranging from to 150 ms. which ind cativatie of this conduction delay. In Lead the PR interval within normal limits, but the ST segment and wave morphology in Lead II do not show significant deviations, suggesting no acute ischemic changes. Leads III and aVF, which reflect the inferior wall, not exhibit pathological waves or significant ST segment changes, ruling out inferiar wall ischemia or infarction. Lead aVi. shows no significant abnormalities in the CRS complex does not show ST elevation, which would
QRS duration slightly prolonged, further supporting the presence of RSBB.
or ST segment, which suggests no high lateral ischemia, have indicated left main coronary artery diseese, In the precordial leads V2-V4, the QRS complexes are significant ST segment clevations depressions anterior wall ischemia. Overall, the EGG findings are consistent with a right bundle branch block, as evidenced the prolonged QRS duration and characteristic patterns in precordial leads. The sinus rhythm and absence of acute ischemic changes suggest that the conduction abnormality the primary finding. This unalysis aligns with the diagresis of abnormal ECG due the presence of RBBB.
widened, consistent with RBBB, and there
suggest
Target Answer (GPT-4o): The lead ECG image revaals key characteristics supporting the diagnosis of right bundle branch block (RBBB)
context of rhythm. Lead the QRS duration extends approximately 134 millsecondss, coupled with notable upslope in the ST segment across multiple heartbeats. The amplitude the QRS complex peaks approsimataly 2.463 mV, which might suggest normal conduction patterns as the load does not display criteria typical of RBBB. Lead V1, however, showrases the classic festures of RBBB with an RaR' pattern evidenced by marginally increased QRS duration and the presence of characteristic Vrrabit ears! morphology, as indicated by significant amplitudes and uniform morphology, sompatible with RBBB diagnostics. Furthermore, different elevations and variations, including notable upslope ST sogmants abserved in leads V1 to V3, can further affirm the presence of bundle branch blocks wi without overt ischeiic patterns Aclditional leads such as II, Ill,,, aVR, and aVF do not exhibit significant anomal les, aside from without inferring signifirant ischamia infarction. The computed everage heart rate is normal, around 73 bpm, with consistent RR intervals suggesting regular complexes and rightward axis deviation are hallmarks of right bundle branch block. These observations, together with consistent acutn ischemic changes are discerned based segment and T wave morphology across most of the ECG leads, correlating with the absence of acute infanction signs. The FCG also appears abnormal due the prolonged QRS complex cluration, supporting the finding of a RBBB when considered in conjunction with the specific electrocardiogrphicc petterns observed.
ST variations, which are consistent across these
sinus rhythm. Although the overall ECG demonstrates sinus rhythm, the broad
measurements, fully support diagnosis
Ground- -ruth Report: Sinus rhythm; Right bundle branch block: Abnormal ECG.
PULSE Interpretation:
The electrocardiggram (ECG) reveals sinus tachycardia, character ized by rapid heart rate with a normal Powave axis, indicating that the sinus node is functioning normally but at an accelerated
pace. Additionally, the ECG shows right bundle branch block (RBBB), which is conduction abnormality right ventricle. Overall, the EGG suggests diagnosis of sinus tachycardia with right bundle branch block.
affecting the right ventricle, resulting delayed activation
Figure 6: Cardiologist Evaluations. Blue: Findings exceeding expert expectations. Yellow: Findings with differing expert opinions.
24

---

25mmas
10mmumy
GEM Interpretation: Upon analyzing the provided ECG image and computed measurements, several key
festures and abnormelities evident. rhythm sinus, as indicated by the regular
preceding each QRS complex. However, the QT interval is prolonged across multiple leads, with QTc values exceeding
500 several instances, suggesting risk for arrhythmias, leads III, and avF, there notable changes consistent with an infor orr infarct, induring abnormal Q waves and ST segment deviations. Additionally. leads V2 to V4 show ST segment elevation and wave inversion, which are indicative of anterior wall ischemia infarction. The lateral leads, particllaly avi, VS, and V6, exhibit changes, such as ST segment depression and T wave inversion, which may suggest lateral myocarcial ischemia. The prasence of multiple leads supports the diagnesis of myccardial ischemia. The combinate nation these findings, including the prolonged QT intervel and the ST-T changes, suggests complex cardiac condition invalving both ischemic and patterns.
changes in
is abnormal, with evidence pointing tawards inferior inforct, lateral ischernia, and a prolonged QT intervel, all of which require further dinical correlation and management.
and
inferior myocardial infarction, as evidenced by significant deviations in the leads primarily reflective of the inferior wall, such as segments
Target Answer (GPT-40): Analyzing the provided 12-lead ECG shows evidence
wave morphology. and ST segment changes suggest suggest an old or acute inferior infarct. Additionally. SI-T changes are notable in the anterior lesc's (V2-V4), indicating anterior or anteroseptal myocardial ischemia. The presence significant elevation or depression, coupled with wave inversion in these leads, corroborates the potential for ischemia in the anterior regions of the heart. In the lateral leads aVi, V5, and V6), the presence of ST segment and wave changes could indicate lateral well ischemia or infarction. Specifically, notable changes in ST form and marphology support this suspicion, which aligns with the observed abnormalities The heart computed at an average of beats per minute, appears normal as shows reguler RR intervals, supporting a sinus thythm without apparent rhythm disturbances. The computed PR intervels and QRS durations in different leads suggest no significant atrioventricular or bundle branch blocks, except for possible localized conduction delays that contribute to the differential ECG pattorns observed. Collectively, these ECG findings demonstrate
loads III, aVF, which
abnormalities
QRS complexes
combination of inferior infarct with the patential ischemia notable in both anterior and lateral regions. This pattern of changes, particularly the ST-T variations across multiple leads, supports the likelihood of broadened myocardial involvement. The diagnosis aligns with classificttio of an abnomal ECG, confirming myncardial ischemia and infarction.
Ground-trun -ruth Report: Sinus rhythm: Inferior infarct: Ant/septal lateral ST-T changes may due myocardial ischemia; Abnormal ECG.
F" PULSE Interpretation:
The electrocardiogrem ogram (ECG) reveals sinus rhythm, indicating normal heart rate and rhythm. However, it also shows an intraventricular conduction clefect, which suggests a delay in the electrical conduction system of the heart, Furthermore, the ECG indicates evidence of previous inforior myocardial infarction, although age
infarct cannot determined, Additionally there are TTT changes observed in the anterior, septal, and lateral leads, which may be indicative of myocardial ischemia. Overall, the ECG is abnormal, suggesting tardiac discase. Clinical Diagnosis: Abnormal ECG with evidence of previous inferior myccardial inforction and possible myocardial schemia.
Figure 7: Cardiologist Evaluations. Blue: Findings exceeding expert expectations. Yellow: Findings
with differing expert opinions.
25

---

25
JCmAvoiV
Upon analyring the provided ECG image and computed mezsurements, several key features and abnormalities are evident. The heart rate is elevated at 101 bpm, indicating sinus tachycardia. In Lead I, the QRS amolitude is notably high, suggesting left ventricular hypertrophy (LVH), which is further Leads V5 and V6. The presence secondary repolanization abnormel ides is suggested by the ST segment changes and wave morphology in these leeds in Lead Il, the P wave amplitude and duration are within normal limits, but the interval is slighlly prolanged, which could indicate first-doge degren AV block. The ORS complexes Leads III and aVF are diminished, which might suggest an inferior wall issue, but lack of significant waves ST segment changes makes this less likely Lead aVL shows a high QRS amplitude, consistent with LVH, and the I wave morphology is abnormal, which could indicate lateral ischemia,, Lead avR does show significant coronary artory disease. In Lead V1, the absence of an raR' pattorn and the lack of significant ST-T changes right bundle branch black is unlikely. Leads V2 to V4 show
GEM Interpretation:
supported by the increased ORS amplitude
elevation, reducing the likelihood of left main
suggest that right ventricular hypertrophy significant waves or ST segment elevation, reducing the likelihood an anterior myocardial inferction. Overall, the ECG findings are consistent with sinus tachycarcia, probable left atrial enlargement, and left ventricular
with secondary repolarization abnormalities. The absence of significant ST segment elevation depression in the anteriar leads suggests that acute ischemnia infarction unlikely. The findings are consistent with chronic chronicchanges changes rather then acute pathology.
Target Answer (GPT-4o): do): In analyzing the ECG image, soveral key features and abnormalities are identifieed that support the clinical findings. The rhythm across leeds is sinus, as observed from the cons stent wave morphology, notably prominent in Lead II, indicating regular atrial rhythm. In Leacl the wave amplitude varies, the durations range around to ms, suggesting left further evidenced by the incressed wave amplitude in several leads, including V1, where the implitude is multiple leads, particularly in V3 and
enlargement. This enlargement is
significantly high. The QRS complexes show increased amplitucles v4, with corresponcing secondary STT changes. This pattern of high voltage QRS complexes and secondary repolarization abnormalities indicative of left ventricular hypertrophy (IVH), The computer measurements confirm increased QRS amplitude in these leads, supporting the presence of LVH. Additionally, there is anterior leads V2-V4, which can attributed the strain pattern associated with LVH rather than acute myccardiial infarction, as correlates with the presence of LVH. There is no significant Q wave development evidence of acute ischemic injury that suggests infraction. Overall, the ECG findings collectively indicate sinus rhythm with left atrial enlargement and left ventricular hypertrophy with secondary repolarization abnomallties. The anterior ST elevation observed is likely consequence of LVH, consistent with LVH strain pattern, These interpretations align with the clinical manifestations of a patient with left heart strain and structural changes, corroborated by the data synthesi from both visual ECG interpretation and detailed interval measurements.
ST segment slevation
Ground truth Report: Sinus rhythm; Left atrial enlargement; LVH with secondary repolarization abnormality: Anterior elevation, probably due to
tachycardia, characterized by rapid rate with normal Pwwave axis, indicating that the sinus node is furitioning normally but at an accelerated pace. Additionally, the ECG shows evidence of biatrial enlargement, suggesting that both the left and right atria are enlarged, which can be sign of cardiac chamber dilation or hypertrophy. Furthermore, the ECG demonstrates left ventricular hypertrophy (LVH) with secondary repolarization abnormality, indicating that the left ventride is thickened. leading to abnormal electrical conduction and repolarization patterns.
PULSE Interpretation: The becttrocrdiingram (ECG) reveals
Figure 8: Cardiologist Evaluations. Blue: Findings exceeding expert expectations. Yellow: Findings with differing expert opinions.
26

---

v5
25mmus
KCmmmny
Upon analyzing the provided ECG image and computed measurements, several key features and abnormallties evident. The rhythm is sinus, as indicated
GEM Interpretation:
consistent wave morphology across leads, but there is a borderline first-degree A-V block, suggested by the prolonged PR intervals, particularly noticeable leads and II. The QRS complexes exhibit right bundle branch block pattern, characterized by a widened ORS duration, particularly lod V1, where an rsf pattern is observed. The axis rightward, which is consistent with the right bundle branch block. In the inferior leads (ll, III, and aVF), there could indicate ischemia or other non specific changes. The amplitude not definitve for
nonspecific: wove changes,
in lead III is notably low, which might suggest an inferior wall issue, although the changes infarction. The lateral leads (l, aVl, V5, and V6) show some wave changes but these are nonspecifii and do not deerrly indicate lateral jschemia inferction. The Qle intervals prolonged several leads, which could concorn for patential arhythmogenic risk. overall
is abnormal, with the combination of these findings suggesting a complex cardiac conduction issue The rightward axis and right bundle branch block are T wave changes the inferior and lateral leads warrant further clinical comrlation and possibly additional diagnostic testing to rule out underlying ischemic heart disease or other cardiac conditions.
significant, and the
Target Answer (GPT-4o): Upon reviewing the 12-lead EGG image, the claggnesis reveals sinus rhythm with rightward axis and the presence a right bundle branch block. The rhythm is consistent with a heart rate around 70 bpm, indicated by relatively regular RR intervals. The rightward axis is evident as lead shows smaller QRS amplituce compared to aVF, which reflects more positive deflections. right buncle perticullll
black is supported by the widened QRS complexes, particularly evident in leads V1 and V2, where raR' pattern could be abserved, consistent with this conduction abnormality ity. Examination the lateral leads avi, V5, V6) reveals mild T wave abnormalities, which are noted as nonspecific changes. In these leads, the morphelogy
shows slight variations that not definitively point ischemia lateral wall changes. Despite these T wave changes being nonspecific they may warrant further clinical correlation given the right bundle branch block already present. summary, the EGG findings primarily indicate a right bundle branch block with a rightward uxis. The wave changes nuted in the leteral leeds are nonspecific and should be clinically correlated seelttedee to assess their significance. These findings characterize the ECG as abnormal, and the conduction delay could suggest underlying pathology that may require additional testing rule out underlying cardiac conditions. This evaluation critical to ensure comprehersive cardiovascular assessment and appropriate subsequent managemeent Ground- -truth Report: Sinus rhythm: Rightward axis; Right bundle branch block Lateral wave changes nonspecific; Abnormal ECG.
suggest some
clinical evaluation
PULSE Interpretation: The electrocard (ECG) reveals sinus mythm, indicating normal heart rate and rhythm. Hawever, the ECG also exhibits several abnormalities, including a prolanged QT interval, which may indicate prolonged repolarization phuse, and right axis deviationi suggesting abnormal orientation of the heart's electrical axis. Ado tionally, the ECG shows signs right ventriculer hypertrophy, low QRS voltages in the limb leads, and extensive ST-T changes, which may be attributed to either hypertrophy or ischemia, loading an overall abnormal ECG.
Figure 9: Cardiologist Evaluations. Blue: Findings exceeding expert expectations. Yellow: Findings with differing expert opinions.
27

---

A.6 Broader Impacts and Limitations
Broader Impacts. This work explores the potential of leveraging large language models generate high-granularity ECG interpretations without the need for manual annotation. It contributes to the medical AI community in three key aspects: First, GEM is the first multimodal large language model that unifies time series, images, and text to enable feature-grounder grounded analysis, evidence- based diagnosis, and clinician-like diagnostic workflows. This opens new avenues for research on multimodal alignment in ECG interpretation. Second, the proposed data generation methodology and resulting Gro-Ginnn dataset establish the first high-resolution resource for grounded ECG understanding. This dataset can support future research aimed developing clinically applicable conversational ECG models. Third, the introduction of the Grounded ECG Understanding task provides comprehensive and fine- grained evaluation protocol, helping future models to be assessed more thoroughly and precisely in terms of clinical reasoning and interpretability. By bridging computational precision with clinician-leve reasoning, GEM represents step toward more reliable, explainable, and clinically applicable Al-assisted ECG analysis, with potential for broader integration into weal-world healthcare workflows.
the other hand, important to acknowledge that the techniques explored in this study are intended for research purposes. Therefore, the proposed GEM model should not be directly used to make critical clinical decisions. Despite its strong performance in ECG interpretation, GEM designed to serve
assistive tool rather than standalone solution for high-stakes clinical use.
Limitations. As discussed in Section 4.4, although our knowledge-guiee instruction data generation approach avoids costly expert annotations while producing high-quality target answers, GPT-4o still occasionally generates responses that may not fully align with cardiologist interpretations. This limitation could be addressed by incorporating expert feedback into the data generation loop, which we identify as an important direction for future work.
28

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The main claims made in the abstract and introduction accurately reflect the paper's contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
The limitations are discussed in Section A.6.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]

---

Justification:
The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: Source codes provided with instructions.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
30

---

reproducible, they Answer: [Yes]
Justification: Source codes are provided with instructions.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (uttps://nips.cm cc/
mublic/guides Saaiidee//CodSuumiii. oooooiicyy for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https:
gguides/CodeSubmii / wtt.ccccppppccccccccc wwwcc//ggge////eee /oodeSubmeeeeeeee woabbbisssoooo assionooliii
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: The full details are provided with the code.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: We have conducted comprehensive evaluation of our proposed methods across wide range of tasks and models. The results consistently demonstrate the effectiveness of our approach. Given the breadth and robustness of these findings, we have chosen not to include formal significance testing.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
31

---

The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative emrr rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We provide these information in Section 3.5.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: : http///eerrp...ee/
Answer: [Yes]
Justification:
This paper adheres to the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer:
[Yes]
Justification:
The broader impacts are discussed in Section A.6.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
32

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: All assets are publicly available and properly cited in the paper.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/dataseta has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
33

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
We provide the documentation along with the code.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/cooe/mmodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question:
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
34

---

Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, or originality the research, declaration is not required.
Answer: [NA]
Justification: The core method development in this research does not involve LLMs any important, original, or non-standard components.
Guidelines:
The answer NA
that the method development in this research does involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy (https : ht/pa/rrps.cccoooeeeeeeeeeeeeeee / for what should or should be described.
35


Reviews:
[{"review_id": "BjkWl3CbD1", "reviewer": "Reviewer_jRz5", "summary": "This paper introduces GEM, designed to provide grounded, clinician-aligned interpretations of electrocardiograms (ECGs) by synergistically using ECG time series data, 12-lead ECG images, and text. The work makes several notable contributions, including a novel data generation pipeline, a new high-granularity dataset, and a clinically-oriented evaluation benchmark.", "questions": "1. Given that LLMs can produce factual inaccuracies, what specific quality control measures, if any, were implemented to validate the clinical accuracy of the generated interpretations? Were any samples reviewed by human experts?\n2. The proposed task is evaluated using GPT-4o, which introduces potential for self-reinforcement and stylistic bias. Can the authors comment on this limitation and the steps taken to ensure the evaluation is as objective as possible?\n3. The data generation method relies on GPT-4o. Could the authors provide an estimate of the computational cost and resources required to generate the 30,000-sample dataset?\n4. The cardiologist evaluation in Appendix A.4 highlights several cases where expert opinions differed from the interpretations generated by GEM or GPT-4o. How do the authors interpret these disagreements?\n5. Can the authors use candle plots to comprehensively reveal the stability of grounded ECG understanding for different models?", "limitations": "1. The primary limitation is the reliance on GPT-4o to generate the ground-truth labels for the ECG-Grounding dataset. As seen in the cardiologist evaluation, GPT-4o is not an infallible medical expert and can produce interpretations that are subtly or overtly incorrect.\n\n2. The use of GPT-4o as the evaluator for the Grounded ECG Understanding benchmark introduces potential biases. The metric may inadvertently reward stylistic similarity to GPT-4o's outputs over pure clinical accuracy. It could be beneficial if the authors could consider metrics that supports the trustworthiness of the LLM judgement.\n\n3. As with any clinical AI tool, there are potential risks. The primary risk is clinical error stemming from the model's recommendations, which could be compounded if the model was trained on flawed synthetic data. Over-reliance on the tool could also lead to \"automation bias,\" where clinicians may accept an incorrect AI suggestion. Therefore more analysis on reliability considerations could benefit the paper convincing the readers.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**:\n- The paper tackles a critical problem in medical AI: the lack of \"grounded understanding\" in automated ECG interpretation. Medical AI requires grounded reasoning to truly help clinician make informed decisions. \n- The proposed pipeline of creating grounded reasoning data could inspire research in scaling up medical AI. The release of the data could be of interest to the medical AI community.\n\n**Weaknesses**:\n\n- GPT-4o is not a board-certified cardiologist, and prone to hallucination and factual inaccuracies, especially in a high-stakes domain like medicine. The dataset, therefore, may contain subtle (or significant) errors, which are then propagated to the trained GEM model. The paper does not mention any quality control procedure on the created data.\n\n- Using GPT-4o to score the model's output is problematic, as \"LLM-as-a-judge\" approach can be both noisy and biased. The evaluation may reward outputs that are stylistically similar to GPT-4o's own generation style rather than those that are most clinically accurate. This introduces a potential for bias and reduces the objectivity of the results for the new benchmark.\n\n- Since the data generation pipeline largely depends on GPT-4o. This raises concerns about the cost, scalability, and reproducibility.", "was_revised": true, "final_justification": "The strengths outweighs weaknesses, so I maintain the positive rating. However, I do believe the current manuscript can be improved regarding GPT judgement's biases, therefore I do not raise the score.", "rebuttal": "Thank you very much for your constructive comments! We are delighted to see that you found the significance of the problem we studied and the value of our proposed methods. We provide detailed responses to your concerns as follows.\n\n>### **Human Evaluation Results**\n\nTo fully address your concerns, we conduct a comprehensive quantitative human evaluation covering three sources of outputs: GPT-4o generated data, Deepseek-R1 generated data, and GEM generated interpretations.   \n\nIn total, 400 ECG-Grounding data (200 from GPT-4o and 200 from Deepseek-R1) and 200 GEM's interpretation are independently reviewed by eight board-certified cardiologists, using seven predefined clinical criteria designed to assess both reliability and usefulness. This unified evaluation protocol allows us to (1) verify the quality of GPT-4o generated training data, (2) test the effectiveness of open-source substitutes, and (3) validate the clinical utility of the GEM model.\n\nWe report the evaluation results in Table 1 below and provide further discussion in the following responses.\n\n***Table 1: Human Evaluation on GPT-4o/Deepseek-R1 Generated Training Data and GEM's Interpretation***\n\n**Reliability metrics**\n|                | Analytical Relevance | Analytical Accuracy  | Analytical Completeness  | \n|----------------|----------------------|----------------------|--------------------------|\n| **GPT-4o**     | 4.7/5                | 4.6/5                | 4.7/5                    | \n| **Deepseek-R1**| 4.8/5                | 4.7/5                | 4.9/5                    | \n| **GEM**        | 4.6/5                | 4.4/5                | 4.6/5                    |\n\n**Usefulness metrics**\n|                | Reasoning Quality | Findings Novelty | Clinical Value | Overall Satisfaction |\n|----------------|-------------------|------------------|----------------|----------------------|\n| **GPT-4o**     | 4.7/5             | 4.4/5            | 4.7/5          | 4.5/5                |\n| **Deepseek-R1**| 4.8/5             | 4.5/5            | 4.6/5          | 4.7/5                |\n| **GEM**        | 4.6/5             | 3.9/5            | 4.3/5          | 4.4/5                |\n\n```\nScoring criteria (Note: Due to space limitations, we only present scores 4–5 here. The full scoring criteria (including 1–3) will be provided in the updated version):\n\nAnalytical Relevance: Do the model’s analyses closely support the diagnosis, and is there corresponding ECG evidence?\n5: Every analysis point is highly relevant to the diagnosis, with clear supporting evidence.\n4: Most analyses are strongly relevant, with minor insufficiencies.\n\nAnalytical Accuracy: Are there any medical factual errors in the model's output?\n5: Completely accurate\n4: Mostly accurate\n\nAnalytical Completeness: Does the model comprehensively discuss key ECG components relevant to the diagnosis, including rhythm, intervals, and waveforms?\n5: All relevant ECG features (rhythm, PR, QRS, ST, T waves, intervals, etc.) are accurately discussed.\n4: Most key ECG features are covered, with minor omissions.\n\nReasoning Quality: Does the model provide a clear, evidence-based reasoning process similar to that of a clinician, logically deriving the diagnosis from ECG features?\n5: Clear and coherent reasoning structure, explaining each step from ECG to diagnosis causally.\n4: Overall reasonable reasoning, but some steps lack detail.\n\nFindings Novelty: Does the model provide insights or findings not noticed by the clinician?\n5: Important new diagnoses or findings.\n4: Novel and somewhat insightful content.\n3: Some new findings, but of limited value.\n\nClinical Value: Does the model output help in clinical decision-making?\n5: Direct and significant support for clinical judgment; content is clear and reliable.\n4: Most content is helpful and practically useful.\n\nOverall Satisfaction: Subjective rating of the overall quality of this analysis.\n5: Very satisfied.\n4: Satisfied\n```\n\n>### **[Weakness 1, Question 1, Limitation 1, Weakness 3, Question 3, Limitation 3] Concerns on GPT-4o generated training data**\n\n- **Data quality control**\n\n    Thank you for raising this important point. We fully agree that quality control is essential when using LLM-generated data in clinical applications. In our data construction pipeline, we incorporated human expert review as a key quality control measure.\n\n    The quantitative human expert evaluation results in Table 1 shows that GPT-4o consistently achieves high scores across both reliability and clinical usefulness, with particularly strong performance in analytical completeness and reasoning quality. These results demonstrate that, with our data generation pipeline, GPT-4o is capable of generating high-quality ECG interpretations that are both clinically reliable and practically valuable.\n\n\n- **GPT-4o cost, scalability, and reproducibility**\n\n    In our implementation, generating each sample required approximately 6,000 input tokens and 300 output tokens, resulting in a total cost of roughly USD $3,000 using the OpenAI GPT-4o API for the 300,000 samples in ECG-Grounding dataset.\n\n    We recognize that GPT-4o, while powerful, is not open-sourced and introduces additional cost and access constraints, which may affect scalability and reproducibility.\n\n\tTo address these concerns, we provide two solutions: \n\n     - First, given the effectiveness of our method, we have scaled up the data generation to the full MIMIC-IV-ECG dataset, which includes 784,680 ECG records from 160,597 patients. These GPT-4o generated grounding data are publicly released and intended to benefit the community as a high-quality and large-scaled resource.\n\n    - Second, to support researchers who wish to apply our method to their own datasets, we test an open-source substitute model, the latest Deepseek-R1. As shown in Table 1, Deepseek-R1 also shown strong capability of generating clinically acceptable, high-quality ECG interpretations with our methods. This demonstrates that our method is adaptable to alternative LLM backbones and remains applicable in settings without commercial API access.\n\n    We will include these results and provide a discussion of the limitations you mentioned in the revised version of the paper.\n\n>### **[Weakness 2, Question 2, Limitation 2] “LLM-as-a-judge” may introduce potential bias**\n\nWe understand the reviewer’s concern regarding potential stylistic bias and self-reinforcement in using GPT-4o as the evaluation model.\n\nTo mitigate these risks and ensure objective assessment, we carefully designed the evaluation prompt to include quantitative scoring criteria and mandatory justification for each score. This structured format helps constrain GPT-4o's evaluation behavior and reduces susceptibility to superficial stylistic similarity.\n\nTo further mitigate potential bias, we plan to include a comparative analysis using multiple independent LLMs (e.g., Deepseek-R1, LLaMA) to evaluate the same test outputs in our updated manuscript. This will enable a more comprehensive assessment of the model outputs and help reduce the risk of bias introduced by relying on a single model for judgment.\n\nIn addition, as in Table 1, our human evaluation on GEM generated interpretations demonstrate that GEM consistently achieves high scores across both reliability and usefulness dimensions, with most metrics rated above 4 out of 5.\n\nThese findings indicate that GEM is capable of producing clinically meaningful and accurate interpretations that align well with cardiologists’ expectations. Also, the human evaluations on GEM are consistent with GPT-4o’s assessment, and support GEM’s potential as a trustworthy assistant for real-world cardiology applications. \n\nWe will add the results and discussions in our revised paper.\n\n>### **[Question 4] Disagreements between cardiologists and models**\n\nThank you for this thoughtful question. We interpret the observed disagreements between cardiologists and model outputs as a reflection of the inherent variability in clinical ECG interpretation.\n\nOverall, the diagnostic conclusions are often aligned between models and human experts. However, discrepancies may arise in finer-grained feature judgments, such as waveform morphology or interval thresholds. This is not unexpected, it mirrors real-world clinical practice, where interpretation differences at the detail level are common even among experienced cardiologists.\n\nImportantly, GPT-4o generated explanations are conditioned on the ground-truth diagnosis from the original clinical reports, which represent a physician’s interpretation at the time of care. As a result, GPT-4o offers one valid expert perspective, while the reviewing cardiologist may offer another. This is consistent with known inter-observer variability, particularly in morphology-sensitive cases. \n\nRather than treating these inconsistencies as errors, we believe they represent valuable opportunities. They highlight cases where model reasoning could be refined with human feedback, and they underscore the importance of modeling expert disagreement to better capture clinical uncertainty. In future work, we plan to explore multi-expert consensus frameworks, inspired by real-world clinical workflows for handling disagreements, such as multi-physician case reviews, to further improve model reliability and alignment with clinical standards.\n\n\n>### **[Question 5] Candle plot for different models**\n\nThank you for the insightful suggestion. While we are unable to include figures during the response period, we plan to incorporate candle plots in the revised paper to illustrate the distribution and consistency of model performance, particularly under evaluation by multiple independent LLMs as previously discussed."}, {"review_id": "uUDzko6Lhp", "reviewer": "Reviewer_W2KZ", "summary": "The authors introduce GEM, an MLLM for 12-lead ECG interpretation that fuses raw time series, rendered ECG images, and text prompts. A dual-encoder architecture aligns signal and image embeddings, which are then used to prompt a GPT-4o model to generate high-granularity instructional data for the LLM. Instruction-tuning on a 30K samples ECG-Grounding dataset lets GEM generate cardiologist-style reports that explicitly refer to waveform evidence. Experiments on a new Grounded-ECG benchmark and on standard classification tasks show reasonable gains in diagnosis accuracy and explanation quality over prior works.", "questions": "1. Did the authors run experiments substituting GPT-4o with an open-source alternative? If yes, what kind of a performance dip can one expect?\n\n2. Did the authors experiment with more LLM decoders? Did swapping those have a significant impact on performance, as observed in the case of JoLT (and BLIP-2)?", "limitations": "Yes.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 4, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. The paper is very well written, the task is well explained and motivated, and the GEM architecture is very well illustrated. The illustrations, in particular, make the paper easy to follow. \n\n2. The authors have done fairly extensive evaluation on eight different metrics, and the ablations justify the use of time series and image input. \n\n3. Strong reproducibility: The authors have made their code public, and the prompts and metrics are very well detailed in the appendix. The ECG-grounding dataset has also been made public.\n\n4. Strong performance gains over Pulse and GPT-4o for both in-domain and out-of-domain ECG-Understanding, and ECG-Bench abnormality detection. Evidence-linked explanations further improve transparency, and the qualitative examples show substantially better explainability than baselines.\n\nWeaknesses:\n\n1. Owing to the utilization of GPT-4o in the pipeline, I would an estimate API cost for training GEM. Similarly, inference time numbers are missing.\n\n2. The architecture, while novel, takes some inspiration from JoLT (Cai et al., 2023), which also does ECG interpretation, which isn't compared and contrasted with in the paper. \n\n3. Although this has been duly noted by the authors, GPT-4o interpretations might occasionally not align with those by experts. This is more critical in the healthcare domain, which GEM is focused on, compared to other domains.", "was_revised": true, "final_justification": "I really appreciate the work put the original submission and the rebuttal, and I appreciate the overall quality of the paper, and the depth of experiments performed. Based on a thorough review of the paper again, including evaluating the novelty embodied in the paper, and the depth of the experiments performed, I believe a rating of 5 is suitable for the paper. Thus, I am going to retain the original score. I applaud the authors for their systematic rebuttal and the overall quality of the paper.", "rebuttal": "Thank you very much for your encouraging comments on our paper! We are glad that you found that the paper is presented and articulated very clearly. We provide detailed responses to your concerns as follows.\n\n>### **[Weakness 1, Question 1] GPT-4o cost, inference time numbers, scalability**\n\n- GPT-4o cost and inference time numbers\n\n    Thank you for raising this important point. In our implementation, generating each sample required approximately 6,000 input tokens and 300 output tokens, resulting in a total cost of roughly USD $3,000 using the OpenAI GPT-4o API for the 300,000 samples in ECG-Grounding dataset. The average inference time per sample is 8s on a single A100 GPU.\n\n- Substituting GPT-4o with an open-source alternative\n\n    We test an open-source substitute model, the latest Deepseek-R1. We conduct a human evaluation on 400 ECG-Grounding data (200 from GPT-4o and 200 from Deepseek-R1).\n    \n    The human evaluation is conducted by eight board-certified cardiologists, each independently evaluating a subset using seven predefined criteria that assess both reliability and usefulness in clinical practice.\n\n    As shown in the human evaluation results in Table 1, Deepseek-R1 is also capable of generating clinically acceptable, high-quality ECG interpretations with our methods. This demonstrates that our method is adaptable to alternative LLM backbones and remains applicable in settings without commercial API access. We will include these results in the revised version of the paper.\n\n    Due to time and resource constraints, we have not yet completed GEM training and evaluation on Deepseek-R1 generated data. We plan to include this experiment in the revised paper to quantify the performance changes. \n\n\n    ***Table 1: Human Evaluation on GPT-4o/Deepseek-R1 Generated Training Data***\n\n    **Reliability metrics**\n    |                | Analytical Relevance | Analytical Accuracy  | Analytical Completeness  | \n    |----------------|----------------------|----------------------|--------------------------|\n    | **GPT-4o**     | 4.7/5                | 4.6/5                | 4.7/5                    | \n    | **Deepseek-R1**| 4.8/5                | 4.7/5                | 4.9/5                    | \n\n    **Usefulness metrics**\n    |                | Reasoning Quality | Findings Novelty | Clinical Value | Overall Satisfaction |\n    |----------------|-------------------|------------------|----------------|----------------------|\n    | **GPT-4o**     | 4.7/5             | 4.4/5            | 4.7/5          | 4.5/5                |\n    | **Deepseek-R1**| 4.8/5             | 4.5/5            | 4.6/5          | 4.7/5                |\n\n    ```\n    Scoring criteria (Note: Due to space limitations, we only present scores 4–5 here. The full scoring criteria (including 1–3) will be provided in the updated version):\n\n    Analytical Relevance: Do the model’s analyses closely support the diagnosis, and is there corresponding ECG evidence?\n    5: Every analysis point is highly relevant to the diagnosis, with clear supporting evidence.\n    4: Most analyses are strongly relevant, with minor insufficiencies.\n\n    Analytical Accuracy: Are there any medical factual errors in the model's output?\n    5: Completely accurate\n    4: Mostly accurate\n\n    Analytical Completeness: Does the model comprehensively discuss key ECG components relevant to the diagnosis, including rhythm, intervals, and waveforms?\n    5: All relevant ECG features (rhythm, PR, QRS, ST, T waves, intervals, etc.) are accurately discussed.\n    4: Most key ECG features are covered, with minor omissions.\n\n    Reasoning Quality: Does the model provide a clear, evidence-based reasoning process similar to that of a clinician, logically deriving the diagnosis from ECG features?\n    5: Clear and coherent reasoning structure, explaining each step from ECG to diagnosis causally.\n    4: Overall reasonable reasoning, but some steps lack detail.\n\n    Findings Novelty: Does the model provide insights or findings not noticed by the clinician?\n    5: Important new diagnoses or findings.\n    4: Novel and somewhat insightful content.\n    3: Some new findings, but of limited value.\n\n    Clinical Value: Does the model output help in clinical decision-making?\n    5: Direct and significant support for clinical judgment; content is clear and reliable.\n    4: Most content is helpful and practically useful.\n\n    Overall Satisfaction: Subjective rating of the overall quality of this analysis.\n    5: Very satisfied.\n    4: Satisfied\n    ```\n\n>### **[Weakness 2] Discussion on JoLT**\n\nThank you for bringing JoLT to our attention, we will include a detailed discussion of JoLT in the revised paper, highlighting similarities and differences in model architecture, supervision strategy, and output format.\n\n>### **[Weakness 3] Disagreement between GPT-4o and experts**\n\nWe recognize that disagreement between GPT-4o-generated interpretations and expert opinions is particularly important in the healthcare domain, where model outputs may directly influence clinical reasoning. We interpret these disagreements not simply as model failures, but rather as reflections of the inherent variability in clinical ECG interpretation.\n\nOverall, the diagnostic conclusions are often aligned between models and human experts. However, discrepancies may arise in finer-grained feature judgments, such as waveform morphology or interval thresholds. This is not unexpected, it mirrors real-world clinical practice, where interpretation differences at the detail level are common even among experienced cardiologists.\n\nImportantly, GPT-4o generated explanations are conditioned on the ground-truth diagnosis from the original clinical reports, which represent a physician’s interpretation at the time of care. As a result, GPT-4o offers one valid expert perspective, while the reviewing cardiologist may offer another. This is consistent with known inter-observer variability, particularly in morphology-sensitive cases. \n\nIn addition, expert evaluation in Table 1 confirms that GPT-4o generated ECG-Grounding data satisfy clinical requirements in the majority of cases. The model consistently receives high scores across both reliability and usefulness, with particularly strong performance in analytical completeness and reasoning quality. These results indicate that, under our knowledge-guided instruction data generation framework, GPT-4o can produce ECG interpretations that are not only clinically relevant but also aligned with the expectations of cardiologists.\n\nWe also view these discrepancies as valuable opportunities: they highlight cases where model reasoning could be refined with human feedback, and they underscore the importance of modeling expert disagreement to better capture clinical uncertainty. In future work, we plan to explore multi-expert consensus frameworks, inspired by real-world clinical workflows for handling disagreements, such as multi-physician case reviews, to further improve model reliability and alignment with clinical standards.\n\n>### **[Question 2] Experiment with more LLM decoders**\n\nThank you for the insightful question. We test GEM using Qwen2-7B as the LLM decoder, and the results on ECG-Bench are included in Table 2 below.\n\nUnlike the performance gains observed in JoLT or BLIP-2 with decoder swapping, we do not observe significant improvements when replacing the Vicuna-7B used in LLavA with Qwen2-7B in our setting. We hypothesize two potential reasons:\n\n- Qwen2-7B is not pretrained with vision-language alignment, unlike models such as LLaVA (based on Vicuna), which may limit its ability to process cross-modal input effectively.\n\n- Qwen2-7B has a significantly larger vocabulary than Vicuna, which may reduce fine-tuning efficiency or generalization under constrained supervision, particularly in specialized domains like ECG interpretation.\n\nIn future work, we plan to explore other vision-language pretrained LLMs as decoders and study their impact on GEM’s performance across tasks.\n\n***Table 2: Experiment with Qwen2-7B decoders***\n\n| Model            | PTB_XL(AUC) | PTB_XL(F1) | PTB_XL(HL) | CODE-15%(AUC) | CODE-15%(F1) | CODE-15%(HL) | CPSC2018(AUC) | CPSC2018(F1) | CPSC2018(HL) | CSN Accuracy | G12EC Accuracy |\n|------------------|------------|-----------|------------|---------------|-------------|--------------|----------------|--------------|---------------|---------------|----------------|\n| GEM - SFT LLavA  | 81.8       | 73.6      | 11.6       | 90.5          | 84.8        | 5.1          | 74.1           | 52.0         | 9.0           | 92.6          | 81.8           |\n| GEM - Qwen2-7B   | 80.8       | 71.9      | 12.4       | 89.3          | 83.1        | 5.2          | 75.0           | 52.8         | 9.1           | 91.1          | 81.1           |"}, {"review_id": "ydmM2fEaNC", "reviewer": "Reviewer_Zvng", "summary": "This paper introduces GEM, a novel MLLM designed for grounded and interpretable ECG analysis. The authors address two key limitations in existing models: the insufficient synergy between ECG time series and image modalities, and the lack of explainability linking diagnoses to specific waveform evidence. To tackle these issues, GEM uniquely integrates a dual-encoder architecture for both time series and images, along with a cross-modal alignment mechanism to enable holistic understanding. A major contribution is the proposal of a knowledge-guided method for automatically generating ECG-Grounding, the first high-granularity instruction dataset that explicitly connects diagnoses to measurable physiological parameters. Furthermore, the paper introduces the Grounded ECG Understanding task, a new clinically-oriented benchmark designed to comprehensively evaluate the model's evidence-based reasoning capabilities.", "questions": "Q1: The Diagnosis Guider is a core component that relies on a predefined set of rules. Could you elaborate on its robustness? Specifically, how does this rule-based system handle rare or complex cardiac conditions that may not be explicitly covered by the existing rules, and what is its potential failure mode in such scenarios?\n\nQ2: The experimental results show a counter-intuitive finding where the GEM variant based on the general-domain LLaVA performs on par with, or even slightly better than, the variant based on the ECG-pretrained PULSE. Could you provide a possible explanation for this? Does this suggest that the pre-training strategy of PULSE might introduce a certain bias that is suboptimal for learning the new evidence-based grounding task you propose?", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "S1. The proposed GEM model is the first to synergistically unify ECG time series, images, and text, establishing a novel multi-modal framework for more comprehensive ECG interpretation.\n\nS2. The creation of the ECG-Grounding dataset via a knowledge-guided generation process provides the high-granularity resource for training and evaluating evidence-based diagnostic models.\n\nS3. The introduction of the Grounded ECG Understanding task offers a new, clinically-oriented benchmark that enables a more thorough assessment of a model's explainability and grounding capabilities beyond traditional metrics.\n\nW1. The paper's parameter-efficient fine-tuning strategy, which freezes the encoders, may prevent the model from optimally adapting its feature extraction to the new grounding task. An exploration of full end-to-end fine-tuning is a notable omission.\n\nW2. The quality of the auto-generated ECG-Grounding dataset is not quantitatively validated by human experts, which is a critical step to ensure its reliability for training and evaluation.\n\nW3. The ablation study lacks an \"IMG-only\" baseline, making it difficult to fully assess the complementarity of the two modalities. Furthermore, the marginal performance gains from adding images in some tasks (Table 2) do not fully support the strong synergy claim.\n\nW4. The paper lacks a systematic analysis of failure cases, making it difficult to pinpoint whether errors stem from faulty feature extraction or flawed logical reasoning within the model.", "was_revised": true, "final_justification": "My concerns have been well-addressed. The authors provided detailed explanations and supplementary experimental results that have resolved my questions. I believe my current rating is appropriate.", "rebuttal": "We appreciate your recognition of the innovation and contributions in our work! We are glad that you found the paper novel and the studied problem is important. We address your concerns point by point as follows. \n\n>### **[Weakness 1] Parameter-efficient fine-tuning may limit model adapting to new grounding task**\n\nThank you for raising this important point. We agree that unfreezing the encoders could potentially improve the model’s ability to adapt feature extraction to the grounding task, particularly in a multimodal setting where fine-grained cross-modal alignment is essential.\n\nHowever, fully training all encoders incurs substantial computational cost, as it requires training two additional Transformer encoders.\nFor this reason, we designed a cross-modal alignment learning module and adopted a widely used parameter-efficient training strategy (e.g., as in LLaVA). This design choice significantly reduced training cost and memory usage while maintaining good performance.\n\nWe appreciate this suggestion and see end-to-end fine-tuning as a valuable next step. We plan to include this experiment in the revised version to better understand its impact on grounding performance.\n\n\n>### **[Weakness 2] Data quality evaluation with human experts**\n\nWe conduct a quantitative human evaluation on 200 unique ECG cases, randomly sampled from ECG-Grounding generated by GPT-4o.\n\nThe evaluation is performed by eight board-certified cardiologists, each independently assessing a subset of cases using seven predefined criteria designed to measure both reliability and clinical usefulness.\n\nTable 1 summarizes the results. The expert evaluation shows that GPT-4o consistently achieves high scores across both reliability and clinical usefulness. These findings indicate that, under our knowledge-guided instruction data generation, GPT-4o can produce ECG interpretations that are both clinically reliable and practically valuable.\n\n***Table 1: Human Evaluation on GPT-4o Generated Training Data***\n\n**Reliability metrics**\n|                | Analytical Relevance | Analytical Accuracy  | Analytical Completeness  | \n|----------------|----------------------|----------------------|--------------------------|\n| **GPT-4o**     | 4.7/5                | 4.6/5                | 4.7/5                    | \n\n**Usefulness metrics**\n|                | Reasoning Quality | Findings Novelty | Clinical Value | Overall Satisfaction |\n|----------------|-------------------|------------------|----------------|----------------------|\n| **GPT-4o**     | 4.7/5             | 4.4/5            | 4.7/5          | 4.5/5                |\n\n\n```\nScoring criteria (Note: Due to space limitations, we only present scores 4–5 here. The full scoring criteria (including 1–3) will be provided in the updated version):\n\nAnalytical Relevance: Do the model’s analyses closely support the diagnosis, and is there corresponding ECG evidence?\n5: Every analysis point is highly relevant to the diagnosis, with clear supporting evidence.\n4: Most analyses are strongly relevant, with minor insufficiencies.\n\nAnalytical Accuracy: Are there any medical factual errors in the model's output?\n5: Completely accurate\n4: Mostly accurate\n\nAnalytical Completeness: Does the model comprehensively discuss key ECG components relevant to the diagnosis, including rhythm, intervals, and waveforms?\n5: All relevant ECG features (rhythm, PR, QRS, ST, T waves, intervals, etc.) are accurately discussed.\n4: Most key ECG features are covered, with minor omissions.\n\nReasoning Quality: Does the model provide a clear, evidence-based reasoning process similar to that of a clinician, logically deriving the diagnosis from ECG features?\n5: Clear and coherent reasoning structure, explaining each step from ECG to diagnosis causally.\n4: Overall reasonable reasoning, but some steps lack detail.\n\nFindings Novelty: Does the model provide insights or findings not noticed by the clinician?\n5: Important new diagnoses or findings.\n4: Novel and somewhat insightful content.\n3: Some new findings, but of limited value.\n\nClinical Value: Does the model output help in clinical decision-making?\n5: Direct and significant support for clinical judgment; content is clear and reliable.\n4: Most content is helpful and practically useful.\n\nOverall Satisfaction: Subjective rating of the overall quality of this analysis.\n5: Very satisfied.\n4: Satisfied\n```\n\n>### **[Weakness 3]  IMG-Only baseline and performance gain from adding images**\n\nThank you for the insightful suggestion. We did not train a separate IMG-only variant of GEM since PULSE can be seen as the IMG-only baseline, as it shares the same configuration used in our ablation study.\n\nThe only difference is that PULSE was trained for three epochs, whereas our ablation variants were trained for one epoch due to resource constraints. Despite this, the TS+IMG still outperforms PULSE in most tasks, suggesting that adding modalities improves both training efficiency and performance, and that the two modalities are complementary. We will clarify this point in the revised version.\n\nIn addition to performance, we believe that adding the image modality also improves usability, for example, by making it easier for users to interact with the system using a photo of their ECG.\n\n>### **[Weakness 4] Failure case analysis**\n\nThank you for this insightful and valuable comment. We conduct a systematic analysis of failure cases, particularly informed by the expert feedback collected during the human evaluation. Based on these observations, we categorize the major failure types and analyze potential causes as follows:\n\n* Incorrect diagnosis\n\n    These errors may result from limitations in the representation stage. For example, certain morphological features such as subtle ST-segment changes or P-wave abnormalities were sometimes missed by the encoders. \n    \n    Potential causes include: \n\n    * Representation deficiency: Some features were not captured due to encoder limitations.\n\n    * Training data limitations: The current training set may not cover the full spectrum of ECG variations, especially when data distribution shifts.\n\n\n* Overstating the severity of findings\n\n    GEM occasionally exaggerates the severity of certain cardiac conditions, which some cardiologists noted could lead to unnecessary patient anxiety.  \n    \n    Potential causes include:\n    \n    * Limited context on patient history: In real-world settings, physicians contextualize ECG findings using patient history, which GEM does not have access to, potentially leading to overdiagnosis in isolation.\n\nIn the revised version, we will include this failure case analysis to provide a more systematic view of where errors arise. These insights will not only contextualize current performance but also inform future directions for improving model safety, reliability, and alignment with clinical practice.\n\n>### **[Question 1] Robustness of Diagnosis Guider and how to handle rare or complex cardiac conditions**\n\nThank you for this important question regarding the robustness of the Diagnosis Guider. We address your concerns as follows:\n\nFirst, rule-based interpretation is the foundation of clinical ECG analysis in current medical practice. For example, the widely adopted AHA/ACCF/HRS guidlines [1] provide structured diagnostic criteria for arrhythmias, conduction delays, and ischemic changes. Commercial ECG systems such as GE MUSE [2] and Philips DXL [3] also rely on deterministic rule sets for generating diagnostic statements. Our Diagnosis Guider is developed with reference to standard clinical guidelines, aiming to analyze ECGs in a way aligned with cardiology practice.\n\nSecond, we acknowledge that rule-based systems, like clinical interpretation itself, can be fallible, especially in rare disease or complex conditions. To improve the overall robustness, our framework does not rely solely on the Diagnosis Guider. It contains learning-based components that can capture broader data-driven patterns. This hybrid design enhances the overall robustness compared to standalone rule-based interpretation. Furthermore, our human evaluation shows that cardiologists found Diagnosis Guider outputs clinically reliable and useful. Together, these results indicate that the system is capable of handling a wide range of ECG interpretation scenarios with performance approaching clinical applicability.\n\nThird, failure modes may still arise, particularly for rare, under-represented, or novel cardiac conditions that fall outside both the current rule base and training data distribution. While we recognize that the system is not yet perfect, the overall performance, both in benchmark evaluations and in expert review, indicates that it has reached a level that helpful in many settings and can provide practical clinical value.\n\nIn the future, we plan to further strengthen the Diagnosis Guider by expanding the rule set with expert input, incorporating more diverse training data, and iteratively refining the framework to improve its coverage and adaptability over time.\n\n[1] Rautaharju, Pentti M., Borys Surawicz, and Leonard S. Gettes. \"AHA/ACCF/HRS recommendations for the standardization and interpretation of the electrocardiogram.\" Journal of the American College of Cardiology 53.11 (2009): 982-991.\n\n[2] GE HealthCare. (2019). Implementing MUSE to Streamline ECG Workflow Case Study.\n\n[3] Philips DXL ECG Algorithm Physician’s Guide.\n\n>### **[Question 2] Explanation for SFT LLaVA’s performance compared to SFT PULSE**\n\nThis finding can be explained by the fact that PULSE was not pretrained on any high-granularity grounding data. As a result, PULSE lacks the necessary capabilities for grounded ECG interpretation. The knowledge embedded during its pretraining provides limited support for the evidence-based grounding task we propose."}, {"review_id": "Nzq7fsXY2r", "reviewer": "Reviewer_KnZY", "summary": "This paper introduces a multimodal LLM for ECG interpretation that supports feature-based analysis, evidence-based reasoning, and a clinician-like diagnostic workflow. Across several datasets, the method outperforms prior PULSE-based models and includes a cardiologist evaluation.", "questions": "1. Generalisability beyond ECG\n\n    Question. Could you either (i) add experiments on at least one additional medical modality (e.g., phonocardiograms, chest-x-ray images, or tabular vitals) or (ii) provide a principled analysis of why your architecture would—or would not—transfer to other signal types?\n\n    Actionable guidance. A small-scale ablation (≤1 additional dataset) or a synthetic test showing that the model can ingest non-ECG inputs would be sufficient.\n\n2. Dependence on GPT-4o-generated training data\n\n    Question. What is the quantitative effect of replacing GPT-4o with (a) a weaker open-source LLM or (b) clinician-curated annotations for the same prompts?\n\n     Actionable guidance. Please report performance when the synthetic corpus is produced by another model (e.g., MedAlpaca, or GPT-3.5) and, if feasible, when a small fraction is manually verified by cardiologists.\n\n3. Validation of GPT-4o's clinical reasoning depth\n\n    Question. Have you benchmarked GPT-4o on tasks closer to specialist reasoning (e.g., Board-style ECG interpretation questions with free-text answers) rather than multiple-choice USMLE items?\n\n    Actionable guidance. Reporting such a benchmark or summarising known literature—and clarifying how any shortcomings might affect label quality—would help contextualise the training pipeline's reliability.", "limitations": "The authors should address the following critical limitation that appears to be missing from their current discussion:\nKnowledge inheritance and error propagation: If there are errors in GPT-4o's knowledge base, there is a possibility that these same errors may also be present in this model. This represents a significant limitation as any systematic biases, factual inaccuracies, or knowledge gaps from the source model could be perpetuated or even amplified in the current system. The authors should explicitly acknowledge this risk and discuss how they have attempted to identify, validate, or mitigate such inherited errors.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "### Strengths\n\n1. Evaluation on multiple datasets shows the proposed approach consistently exceeds baseline methods.\n\n2. Qualitative assessment by cardiologists corroborates the model's effectiveness.\n\n3. The prompts used are provided, enhancing reproducibility.\n\n### Weaknesses\n\n1. The work is confined to ECG data; it lacks experiments or discussion on extending the method to other modalities or on the limitations imposed by this narrow focus. While this may suffice for a medical-domain venue, the broader audience of the present conference likely expects such analysis.\n\n2. GPT-4o is employed to create training data, yet its medical knowledge has not been evaluated here. Although GPT-4o scores highly on the USMLE—multiple-choice questions aimed at licensure—this study tackles tasks involving specialist reasoning, and the gap is non-trivial. An empirical assessment or a discussion in the limitations section is needed. It would also help to report how performance changes when data are generated and evaluated with models other than GPT-4o.", "was_revised": true, "final_justification": "The authors have adequately addressed the concerns raised in the initial review. \nThe clarifications provided, particularly regarding generalization beyond ECG and dependence on GPT-4o generated training data, strengthen the manuscript significantly.", "rebuttal": "Thank you very much for your thoughtful comments! We are glad you recognized the value of our method and appreciated the open-source contributions in our work. We respond to your comments in detail below.\n\n>### **[Weakness 1, Question 1] Generalization beyond ECG**\n\nWe appreciate the reviewer’s suggestion to consider generalization beyond ECG. While our work is intentionally focused on ECG, a modality that is not only foundational in cardiovascular diagnostics but also highly accessible and widely adopted across care settings, we agree that exploring generalizability is valuable.\n\nFollowing your suggestion, we conduct experiments extending our methods to respiratory spirogram data for pulmonary disease detection. The dataset was extracted from UK Biobank field 3066. As shown in the Table 1 below, our approach outperforms both deep learning SOTA and LLaMA3.1-8B across multiple metrics. These results suggest that our methods are not inherently limited to ECG and hold promise for broader clinical modalities.\n\nWhile these results demonstrate the potential generalizability of our framework, this paper is intentionally scoped to ECG interpretation. All modeling components and instruction data were designed around the specific structure and diagnostic reasoning challenges associated with ECG data. We believe ECG offers a rich and clinically meaningful testbed for studying multimodal reasoning in depth.\n\nIn the revised version, we will add a discussion on how our framework could be adapted to other medical modalities. This primarily involves adjusting the knowledge-guided instruction data generation pipeline to account for modality-specific representations and clinical reasoning structures.\n\n***Table 1: Experiment results on respiratory spirogram data for pulmonary disease detection***\n\n| Model              | AUROC  | AUPRC  | Sensitivity  | Specificity  | F1 score |\n|--------------------|--------|--------|--------------|--------------|----------|\n| Deep Learning SOTA | 82.7   | 80.7   | 69.1         | 79.0         | 70.1     |\n| Llama3.1-8B        | 76.9   | 73.2   | 98.4         | 12.5         | 64.2     |\n| Ours               | 89.8   | 90.5   | 83.3         | 67.0         | 74.4     |\n\n\n>### **[Weakness 2, Question 2] Dependence on GPT-4o generated training data**\n\nThank you for raising this important point. \n\nTo fully address your concern, we conduct a structured human evaluation covering three sources of outputs: GPT-4o generated data, Deepseek-R1 generated data, and GEM generated interpretations.\n\nIn total, 400 ECG-Grounding data (200 from GPT-4o and 200 from Deepseek-R1) and 200 GEM's interpretation were independently reviewed by eight board-certified cardiologists, using seven predefined clinical criteria designed to assess both reliability and usefulness. This unified evaluation protocol allows us to (1) verify the quality of GPT-4o generated training data, (2) test the effectiveness of open-source substitutes, and (3) confirm the clinical utility of the GEM model.\n\nWe report the evaluation results in the Table 2 below. The expert evaluation shows that GPT-4o consistently achieves high scores across both reliability and clinical usefulness, with particularly strong performance in analytical completeness and reasoning quality. These results demonstrate that, when using our knowledge-guided instruction data generation, GPT-4o is capable of generating high-quality ECG interpretations that are both clinically reliable and practically valuable.\n\nDeepseek-R1 is also capable of generating clinically acceptable, high-quality ECG interpretations with our methods. This demonstrates that our method is adaptable to alternative LLM backbones and remains applicable in settings without commercial API access. We will include these results in the revised version of the paper.\n\nDue to time and resource constraints, we have not yet completed GEM training and evaluation on Deepseek-R1 generated data. We plan to include this experiment in the revised paper to quantify the performance changes. \n\n***Table 2: Human Evaluation on GPT-4o/Deepseek-R1 Generated Training Data and GEM's Interpretation***\n\n**Reliability metrics**\n|                | Analytical Relevance | Analytical Accuracy  | Analytical Completeness  | \n|----------------|----------------------|----------------------|--------------------------|\n| **GPT-4o**     | 4.7/5                | 4.6/5                | 4.7/5                    | \n| **Deepseek-R1**| 4.8/5                | 4.7/5                | 4.9/5                    | \n| **GEM**        | 4.6/5                | 4.4/5                | 4.6/5                    |\n\n**Usefulness metrics**\n|                | Reasoning Quality | Findings Novelty | Clinical Value | Overall Satisfaction |\n|----------------|-------------------|------------------|----------------|----------------------|\n| **GPT-4o**     | 4.7/5             | 4.4/5            | 4.7/5          | 4.5/5                |\n| **Deepseek-R1**| 4.8/5             | 4.5/5            | 4.6/5          | 4.7/5                |\n| **GEM**        | 4.6/5             | 3.9/5            | 4.3/5          | 4.4/5                |\n\n```\nScoring criteria (Note: Due to space limitations, we only present scores 4–5 here. The full scoring criteria (including 1–3) will be provided in the updated version):\n\nAnalytical Relevance: Do the model’s analyses closely support the diagnosis, and is there corresponding ECG evidence?\n5: Every analysis point is highly relevant to the diagnosis, with clear supporting evidence.\n4: Most analyses are strongly relevant, with minor insufficiencies.\n\nAnalytical Accuracy: Are there any medical factual errors in the model's output?\n5: Completely accurate\n4: Mostly accurate\n\nAnalytical Completeness: Does the model comprehensively discuss key ECG components relevant to the diagnosis, including rhythm, intervals, and waveforms?\n5: All relevant ECG features (rhythm, PR, QRS, ST, T waves, intervals, etc.) are accurately discussed.\n4: Most key ECG features are covered, with minor omissions.\n\nReasoning Quality: Does the model provide a clear, evidence-based reasoning process similar to that of a clinician, logically deriving the diagnosis from ECG features?\n5: Clear and coherent reasoning structure, explaining each step from ECG to diagnosis causally.\n4: Overall reasonable reasoning, but some steps lack detail.\n\nFindings Novelty: Does the model provide insights or findings not noticed by the clinician?\n5: Important new diagnoses or findings.\n4: Novel and somewhat insightful content.\n3: Some new findings, but of limited value.\n\nClinical Value: Does the model output help in clinical decision-making?\n5: Direct and significant support for clinical judgment; content is clear and reliable.\n4: Most content is helpful and practically useful.\n\nOverall Satisfaction: Subjective rating of the overall quality of this analysis.\n5: Very satisfied.\n4: Satisfied\n```\n\n\n>### **[Question 3] Validation of GPT-4o's clinical reasoning depth**\n\nDirectly using GPT-4o for board-style ECG interpretation tasks that require specialist-level reasoning typically yields suboptimal results. As shown in Table 3 of our paper, when prompted to generate free-text clinical reports on the PTB-XL dataset, GPT-4o achieves a report score of only 50.2, significantly lower than our method (67.1). This task requiring comprehensive interpretation and is close to to board-style questions. Moreover, GPT-4o also performs poorly on multiple-choice diagnostic tasks (as shown in main paper Table 2 ), highlighting its difficulty in making correct diagnosis even under simplified settings. This further underscores the challenge it faces in generating accurate and complete reasoning in free-text ECG interpretation.\n\nThis is the reason we designed the knowledge-guided instruction data generation, where the GPT-4o is used as a controlled generation tool to construct training data, guided by ground-truth diagnostic reports and heartbeat-level descriptions. With our method, GPT-4o is able to generate high-quality reasoning data. This is supported by human evaluation on GPT-4o outputs, where cardiologists judged the reasoning to be clinically acceptable in most cases.\n\n\n>### **[Limitation 1]  Knowledge inheritance and error propagation risk**\n\nWe appreciate the reviewer’s insightful comment regarding the risk of inherited errors from GPT-4o.\n\nWe agree that if GPT-4o’s factual understanding or clinical knowledge is flawed, these errors could propagate into the generated training data. To mitigate this risk, we designed our data generation pipeline to anchor GPT-4o’s outputs to ground-truth clinical labels. Specifically, GPT-4o is prompted to explain or elaborate on known diagnoses and lead-level features from expert-labeled datasets such as MIMIC-IV-ECG. This constrains the model to generate reasoning that reflects verified clinical diagnosis, rather than freely recalling potentially unreliable knowledge.\n\nOur human evaluation on both GEM’s interpretation and GPT-4o generated training data verified that our data generation pipeline help substantially mitigate the risk of major factual inaccuracies. In future work, we plan to integrate clinical validation tools and expert feedback loops to further improve the reliability and safety of model-generated content.\n\nWe will explicitly include this discussion in the revised manuscript."}]

Author rebuttal:

Meta-review: This paper introduces GEM, a multimodal LLM for grounded ECG interpretation that integrates ECG time series, images, and text. Key contributions include a dual-encoder framework, a knowledge-guided data generation pipeline producing ECG-Grounding, and a new Grounded ECG Understanding benchmark. The work is well-motivated, reproducible, and supported by extensive evaluation, including cardiologist review. The main concerns were dependence on GPT-4o for data generation/evaluation, limited ablations, and lack of discussion on generalizability. The rebuttal convincingly addressed these through human expert validation, substitution experiments with DeepSeek-R1, cost/latency reporting, spirogram generalization, and additional failure case analysis. Reviewers converged on acceptance after the discussion, with some residual caution regarding LLM reliance. 

The paper offers a substantial and well-validated contribution in multimodal medical AI, though the dependency on GPT-4o and limited end-to-end fine-tuning experiments temper its suitability for spotlight/oral.
Decision: Accept (poster)