Teach Multimodal LLMs to Comprehend Electrocardiographic Images

## Abstract

Abstract The electrocardiogram (ECG) is an essential non-invasive diagnostic tool for assessing cardiac conditions. Existing automatic interpretation methods suffer from limited generalizability, focusing on a narrow range of cardiac conditions, and typically depend on raw physiological signals, which may not be readily available in resource-limited settings where only printed or digital ECG images are accessible. Recent advancements in multimodal large language models (MLLMs) present promising opportunities for addressing these challenges. However, the application of MLLMs to ECG image interpretation remains challenging due to the lack of instruction tuning datasets and well-established ECG image benchmarks for quantitative evaluation. To address these challenges, we introduce ECGInstruct , a comprehensive ECG image instruction tuning dataset of over one million samples, covering a wide range of ECG-related tasks from diverse data sources. Using ECGInstruct , we develop PULSE , an MLLM tailored for ECG image comprehension. In addition, we curate ECGBench , a new evaluation benchmark covering four key ECG image interpretation tasks across nine different datasets. Our experiments show that PULSE sets a new state-of-the-art, outperforming general MLLMs with an average accuracy improvement of 15% to 30%. This work highlights the potential of PULSE to enhance ECG interpretation in clinical practice.

## 1 Introduction

The electrocardiogram (ECG) is an essential tool in diagnosing cardiovascular diseases due to its non-invasive, cost-effective, and widely accessible nature for assessing cardiac function. While some approaches have been proposed for automatic ECG diagnosis (Hannun et al., ; Ribeiro et al., ; Hughes et al., ), these are primarily designed for classification tasks with limited cardiac conditions, often lacking generalizability. Moreover, they typically treat ECG data as time-series physiological signals, which may not always be available, particularly in resource-constrained settings where only printed or digital images are accessible (Sangha et al., ; ).

Recent advancements in multimodal large language models (MLLMs) have shown impressive success across vision-language tasks, offering new possibilities for addressing the limitations of traditional ECG models. However, applying MLLMs to ECG interpretation is not straightforward. As illustrated in Fig. 1, current MLLMs, such as GPT-4o (OpenAI, ), often provide responses that appear correct and contextually relevant but are ultimately inaccurate in interpreting ECG images. This highlights the need for specialized MLLMs for ECG image interpretation.

Developing MLLMs for ECG images faces several challenges. First, no large-scale ECG image datasets are currently available as most ECG datasets contain only raw signal data, which needs to be synthesized into digital images. Second, there is a lack of instruction tuning datasets for ECG images. Large high-quality instruction tuning datasets, which are crucial for MLLM development, need to be curated from scratch for ECG-related tasks. Finally, evaluation is just as critical as model development, yet no established benchmark exists for assessing MLLM performance in ECG image interpretation. A well-defined benchmark is essential for both quantifying model performance and identifying areas for future improvement.

In this paper, we tackle these challenges by introducing ECGInstruct, the first large-scale ECG image instruction tuning dataset containing over one million ECG image-text samples. ECGInstruct is characterized by: 1) realistic image synthesis that replicates artifacts commonly seen in paper-based ECGs, 2) a diverse range of ECG-related tasks refined with insights from clinical experts, and 3) data sourced from distinct geographic regions. Leveraging ECGInstruct, we develop PULSE, an MLLM for ECG image comprehension. To evaluate PULSE, we present ECGBench, a comprehensive evaluation benchmark covering four ECG image interpretation tasks across nine different datasets. ECGBench includes repurposed tasks (e.g., abnormality detection) from existing datasets, and newly curated, more challenging tasks using real-world ECG images.

Evaluated on ECGBench, PULSE sets a new state-of-the-art, significantly outperforming proprietary MLLMs across all benchmarks with an average accuracy gain of 15% to 30% compared to GPT-4o on out-of-domain datasets (Fig. 1). Ablation experiments demonstrate the importance of incorporating diverse data sources and ECG instruction tasks into the training data. A case study and discussion further illustrate the model’s effectiveness in ECG image interpretation.

To summarize, our main contributions are as follows,

- •
Problem. We investigate the capabilities of MLLMs in ECG image interpretation and evaluate their performance across various downstream tasks. To the best of our knowledge, this is the first study focused on assessing MLLMs in image-based ECG interpretation.
- •
Dataset. We construct ECGInstruct, a large-scale ECG image instruction tuning dataset consisting of a wide range of ECG-related tasks, serving as a valuable resource for fine-tuning MLLMs for ECG image interpretation.
- •
Model. We develop PULSE, a new MLLM tailored for ECG image interpretation. The model achieves state-of-the-art performance, outperforming both proprietary and open-source MLLMs.
- •
Evaluation. We establish ECGBench, a comprehensive benchmark for evaluating ECG image interpretation, which includes diverse evaluation tasks, both real-world and synthesized images.

Figure: Figure 2: ECGInstruct: a list of diverse and large-scale instruction tuning datasets for ECG image interpretation. (1) ECG images are synthesized from raw signal recordings with various distortions that mimic real-world printed ECG images. (2) ECGInstruct is curated based on clinician-defined ECG-related tasks, original diagnosis and clinical reports, and diverse task types. Additional quality checking is applied to filter lower-scored instructions.
Refer to caption: x2.png

## 2 ECGInstruct: Teach MLLMs to Comprehend ECG Images

We aim to curate a list of multifaceted instruction tuning datasets for ECG analysis that are featured by 1) realistic image synthesis resembling the artifacts in paper ECGs, 2) diverse types of ECG-related tasks with clinical experts’ insights, and 3) different data sources from distinct geographical regions. We show the construction of ECGInstruct in Fig. 2 and data summary in Table 1.

**Table 1: Summary of ECGInstruct. Feature: basic feature recognition, Rhythm: heart rhythm analysis, Morphology: morphology and pathology identification, Report: clinical report generation. Close: close-ended QA, Open: open-ended QA, Fill: fill-in-the-blank, MCQ: multi-choice QA. The full table of data statistics is provided in Appendix Table A1.**
| Source Dataset | Task | Type | # Samples |
| --- | --- | --- | --- |
| PTB-XL (Wagner et al., ) | Feature | Close/Open/Fill/MCQ | 30K |
|  | Rhythm | Close/Open/Fill/MCQ | 36K |
|  | Morphology | Close/Open/Fill/MCQ | 67K |
|  | Report | Open | 16K |
| ECG-QA (Oh et al., ) | Feature | Close | 40K |
|  | Rhythm | Close | 9K |
|  | Morphology | Close | 90K |
| MIMIC-IV-ECG (Gow et al., ) | Feature | Close/Open/Fill/MCQ | 29K |
|  | Rhythm | Close/Open/Fill/MCQ | 115K |
|  | Morphology | Close/Open/Fill/MCQ | 169K |
|  | Report | Open | 487K |
| CODE-15% (Ribeiro et al., ) | Feature | Close | 22K |
|  | Rhythm | Close | 14K |
|  | Morphology | Close | 31K |
| Total (ECGInstruct) |  |  | 1.2M |

### 2.1 ECG Image Synthesis with Various Distortions

To enhance the robustness and real-world applicability of our model, we synthesize ECG images mimicking common artifacts found in paper ECGs. We adopt an ECG image synthesis tool (Shivashankara et al., ) that provides various imperfections such as grid line interference, creases, wrinkles, paper rotations, etc. By including these synthesized artifacts, we aim to train models that can effectively interpret ECGs in less-than-ideal conditions, as often encountered in clinical settings. More details are provided in Appendix C.

### 2.2 ECG-related Tasks with Clinical Experts’ Insights

To construct a comprehensive set of ECG-related tasks, we consulted domain experts to curate diverse and clinically relevant tasks covering four different categories. Each category is designed to address specific aspects of ECG interpretation and analysis, including (1) basic feature recognition (see examples in Appendix Fig. A1), (2) heart rhythm analysis (see examples in Appendix Fig. A2), (3) morphology and pathology identification (see examples in Appendix Fig. A3) and (4) clinical report generation (see examples in Appendix Fig. A4). Basic feature recognition (e.g., interval or segment, etc.) forms the foundation of ECG interpretation, enabling the model to grasp essential cardiac parameters. Heart rhythm analysis (e.g., arrhythmias, conduction abnormalities, etc.) and morphology and pathology identification (e.g., wave shape, pathological conditions, etc.) are more advanced and critical aspects of ECG analysis, ensuring that the model can detect and classify complex conditions accurately. Lastly, clinical report generation mirrors the process of healthcare professionals synthesizing a comprehensive interpretation of an ECG. By incorporating clinical experts’ insights, we encourage the model to learn the practical skills required in a clinical context.

### 2.3 Diverse Types of Tasks and Data Sources

Based on the original diagnoses and clinical reports from the existing ECG datasets, we curate diverse types of tasks including multi-choice questions, fill-in-the-blank, close-ended QA, and open-ended QA.
This variety of task types not only enhances the model’s versatility but also mimics the diverse cognitive processes involved in real-world ECG interpretation. By incorporating these varied task types, we aim to develop a more robust and adaptable model capable of handling a wide spectrum of ECG-related queries and analyses.

To ensure broad applicability and generalizability, we collect ECG data from four different sources across geographically distinct regions: 1) PTB-XL (Wagner et al., ): a Germany-based, publicly available repository; (2) MIMIC-IV-ECG (Gow et al., ): a large set of ECGs for patients who appear in the MIMIC-IV Clinical Database from Beth Israel Deaconess Medical Center in Boston (Johnson et al., );
3) CODE-15% (Ribeiro et al., ): an ECG dataset from a central ECG repository from Minas Gerais, Brazil under the clinical outcomes in digital electrocardiology (CODE) study (Ribeiro et al., ); 4) ECG-QA (Oh et al., ), a question answering dataset for ECGs that is constructed based on PTB-XL (Wagner et al., ). This diverse geographical representation enhances the model’s ability to generalize across different populations and healthcare systems, accounting for potential variations in ECG patterns and interpretations across regions.

### 2.4 Data Synthesizing at Scale

Since large-scale annotation of ECG features is extremely expensive and time-consuming, we develop an automatic data synthesizing pipeline to address this data scarcity issue. We utilized clinical reports from PTB-XL and MIMIC-IV-ECG as initial seed data and leveraged an advanced LLM (i.e., Llama-3-70B-Instruct) for data synthesis. Building upon the expert-in-the-loop process and diverse data resources described in the previous sections, we synthesized a substantial volume of ECG-related instructions and corresponding responses. These were based on expert-provided examples and real-world scenarios, with the specific prompts used in this process detailed in the Appendix E.
For datasets lacking comprehensive reports, such as CODE-15%, we manually constructed diverse templates to transform the existing data into an instruction-response format.

### 2.5 Quality Control

To guarantee the quality of generated instructions and corresponding responses, we apply an independent LLM as a judge to evaluate and score the content. This process involves several steps: 1) initial generation: instructions and responses are first generated using our primary model; 2) evaluation criteria: we establish a set of evaluation criteria including the instruction relevance, clarity, answerability of the responses, etc; 3) LLM judge and scoring: an independent LLM (Llama 3 (Meta, )) is used as a judge to assess each instruction-response pair against established criteria and assign scores (see prompt in Appendix Fig. A8); 4) feedback loop: low-scoring items are flagged for human expert review and potential revision or removal; 5) iterative refinement: based on the scoring patterns and human expert input, we continually refine our instruction generation process. By combining automated LLM evaluation with human expert oversight, we create a robust system for maintaining and improving the quality of our instruction-response pairs.

### 2.6 Training

Our model architecture closely follows that of LLaVA (Liu et al., ; ), adapting it for ECG image analysis. We use a vision encoder to process ECG images and a large language model as the text decoder, connected via a projection layer.
We organize the data into three components: the image, the instructions, and the outputs. The instruction is query or task related to the ECG image and the output is the expected response or prediction base on the image and instruction. We place the image at the beginning of each conversation, serving as the visual grounding for the entire dialogue.
During training, we freeze the parameters of the vision encoder while updating the parameters of the projection layer and the language model using an autoregressive training objective, where we mask all the tokens belonging to the image and the instruction.

Figure: Figure 3: The data curation process for ECGBench. There are four key tasks involved: (1) two repurposed tasks (abnormality detection and report generation) derived from existing ECG datasets, where ECG images are synthesized from raw signals, and queries/answers are extracted based on diagnostic and clinical reports; (2) Two newly developed tasks using external resources, where ECG images and associated questions and answers are collected and generated from real-world sources.
Refer to caption: x3.png

## 3 ECGBench

In this section, we present ECG-Bench (Fig. 3), a comprehensive benchmark for evaluating MLLMs on ECG image interpretation. Our benchmark contains both repurposed tasks from six existing datasets and newly created tasks from external resources. Table 2 shows the details of each evaluation dataset. We introduce the detailed evaluation task curation process below.

Table: Table 2: Overview of evaluation datasets in ECGBench. This collection contains both in-domain and out-of-domain problems across four key tasks with diverse answer types.

### 3.1 Evaluation Task Curation

Abnormality Detection. This task focuses on detecting cardiac abnormalities using ECG images. We curate this task by repurposing six existing ECG datasets: three in-domain datasets: PTB-XL (Super) (Wagner et al., ), CODE-15% (Ribeiro et al., ), ECG-QA (Oh et al., ), and three out-of-domain datasets: CPSC 2018 (Liu et al., ), CSN (Zheng et al., ; ) and G12EC (Liu et al., ). For all datasets, we first synthesize images using raw signals and then curate queries based on the original diagnosis and reports. For datasets with fewer than 10 diagnostic labels, we curate close-ended questions. For those with more labels, we construct multi-choice questions with 8 options, including the original diagnosis and randomly sampled negative labels.

Report Generation. This task involves generating detailed reports for given ECG images. We benchmark using 500 randomly selected reports from the test set of PTB-XL, which contains high-quality ECG reports written and validated by cardiologists. Similarly, the ECG images are synthesized from the raw signals. For the ground truth reports written in non-English (PTB-XL is a Germany-based dataset), we translate the reports into English before the evaluation.

MMMU ECG. Inspired by MMMU (Yue et al., ), a widely adopted evaluation benchmark for MLLMs, we manually curated an ECG version with 200 multi-choice questions with the help of medical school students. The curation process involved three key steps:
(1) Resource Selection: We gathered ECG materials from diverse and reliable sources such as ECG textbooks, clinical case reports from medical journals, and widely used online ECG learning materials. This ensures the comprehensiveness and quality of collected ECG examples and interpretations.
(2) Question Creation and Collection: Five medical school students with basic knowledge of ECG were recruited for this task. They extracted existing questions from the collected resources. For ECG images accompanied only by clinical interpretations, the annotators created questions based on these interpretations. Additionally, they formulated new questions drawing from their expertise, ensuring a balance between various ECG interpretation aspects (e.g., rhythm analysis, morphology assessment, clinical interpretation).
(3) Quality Control: To maintain high standards, we implemented a quality control process. In particular, Each question underwent review by at least two other annotators, checking for accuracy and clarity. An independent reviewer cross-checked the final images, questions, and corresponding answers against the original sources to ensure fidelity to the source material. Any discrepancies or ambiguities were resolved during this process.

ECG Arena. To assess the model’s instruction-following ability in ECG comprehension, we developed ECG Arena, inspired by MT-Bench  (Zheng et al., ) and Arena-hard  (Chiang et al., ) used in general LLM chat evaluations. We manually curated 50 multi-turn ECG-related questions, focusing on open-ended interactions. The data curation process for ECG Arena, like MMMU ECG, involves three main steps: resource selection, question creation, and quality control. The key distinction is that MMMU ECG focuses on multiple-choice questions, whereas ECG Arena involves more complex, flexible multi-turn, open-ended questions. Each follow-up question is contingent on the initial question and its response, making the process more challenging and reflective of real-world applications. Since multi-turn conversations are rare in existing sources, this posed significant challenges during data curation. To address this, annotators created such conversations by referencing original clinical interpretations and ECG images. The questions are designed to feel natural and simulate a real clinical setting (e.g., the first question may ask about basic findings from the image, followed by a question about potential clinical causes or diagnoses based on those findings).

### 3.2 Evaluation Metrics

Abnormality Detection: We use macro AUC, macro F1, and hamming loss (HL) for multi-label datasets, and accuracy for others. Report Generation: We employ GPT-4o as a judge, evaluating reports based on rhythms, waveform, and diagnosis, with a maximum score of 100 points (see evaluation prompt in Appendix Fig. A9).
MMMU ECG: We use accuracy as the primary metric, with systematic, rule-based evaluation pipelines to ensure consistent scoring.
ECG Arena: GPT-4o assesses model performance by comparing generated responses with ground truth answers, considering accuracy, completeness, and instruction adherence, with a maximum score of 100 points (see evaluation prompt in Appendix Fig. A10). More evaluation details are provided in the Appendix F.

## 4 Experiments

### 4.1 Methods for Comparison

In order to evaluate the performance of our proposed model, we compare it against a set of established methods including domain-specific methods and state-of-the-art MLLMs.

- •
Domain-specific Methods: We consider four domain-specific methods for ECG including three signal-based methods: METS (Li et al., ), MERL (Liu et al., ), ST-MEM (Na et al., ), and one image-based method: ECG-GPT (Khunte et al., ).
- •
Proprietary MLLMs: We consider three proprietary MLLMs: GPT-4o, GPT-4o mini (OpenAI, ), Gemini 1.5 Pro (Reid et al., ),
and Claude 3.5 Sonnet (Anthropic, ).
- •
Open-source MLLMs:
We select a range of open-source models to ensure comprehensive coverage across different model sizes and visual components, including the general models LLaVA-1.5 (Liu et al., ; ), LLaVA-1.6 (Liu et al., ), Phi-3-Vision Abdin et al. (), Idefics2-8B (Laurençon et al., ), DeepSeek-Vl-7B (Lu et al., ), Mantis-8B-siglip-Llama3 (Jiang et al., ), MiniCPM-V-2.6 (Yao et al., ), InternVL2 (Chen et al., ; ) and state-of-the-art multimodal models LLaVA-OneVision (Li et al., ), Qwen2-VL (Wang et al., ), as well as the domain-specific models LLaVA-Med (Li et al., ).

### 4.2 Implementation Details

We follow the architecture of LLaVA-v1.6-Vicuna-7B, which includes three core components: a vision encoder, a large language model, and a projector to align image and text modalities. We format all datasets into a chatbot-style multi-turn dialogue format and use the special token “$<$image$>$” to represent image features within the text data. We utilize anyres to support the model’s ability to recognize ECG images of various sizes that may appear in real-world scenarios. We freeze the parameters of the vision encoder and fine-tune all parameters of the projector and LLM. We use a learning rate of 2e-5, set the batch size to 128, and employ a cosine scheduler with a 5% warm-up period for three epochs.

### 4.3 Main Results

We show in-domain the out-of-domain results in Table 3 and Table 4 respectively. Overall, we observe that PULSE achieves state-of-the-art performance on different datasets and tasks.

Results on In-domain datasets.
As shown in Table 3, PULSE demonstrates significant improvements over both proprietary and open-source MLLMs across all in-domain datasets. Specifically, PULSE surpasses the best proprietary model (GPT-4o) with a 27% improvement in AUC, an 11-point gain in report score, and a 39% increase in accuracy on the PTB-XL Super, PTB-XL Report, and ECG-QA tasks, respectively. Moreover, PULSE achieves notable gains over the best open-source model, with a 28% improvement in AUC, a 12-point gain in report score, and a 44% increase in accuracy on the same tasks.

These results highlight the complexity of ECG image interpretation, a task where even the best proprietary models perform near randomly. By fine-tuning on ECGInstruct, PULSE achieves substantial performance improvements, demonstrating the importance of high-quality and task-related instruction tuning. Moreover, while certain domain-specific methods (e.g., MERL) achieve comparable performance on specific datasets, their specialized designs limit their generalization to other diverse tasks, restricting their broader applicability in real-world, complex healthcare scenarios.

Table: Table 3: In-domain evaluation results. ^† indicates results from original papers, ^∗ denotes results obtained using the provided online software, N/A indicates methods not applicable or not designed for certain tasks, and - indicates unreported scores in original papers. Note that the setup of some domain-specific methods is not the same as ours, thus the results listed are for reference purposes.

Results on Out-of-domain datasets.
Table 4 presents the comparison results on out-of-domain datasets, where PULSE consistently delivers outstanding performance. Notably, it achieves a significant 15% improvement in accuracy on the MMMU ECG benchmark compared to GPT-4o. This substantial improvement indicates the PULSE’s robustness and ability to generalize to unseen data.

The ECG Arena benchmark presents a significantly more challenging task for all models. This benchmark is characterized by its multi-turn, open-ended question-answering format, which closely simulates real clinical scenarios. Despite these challenges, PULSE still surpasses the best proprietary model by 2 points and outperforms the leading open-source model by an impressive 11 points in terms of arena score. These results highlight PULSE’s relative strength in handling complex, clinically-oriented ECG interpretation and analysis. Additionally, the performance gap across models on this challenging benchmark indicates considerable room for future improvements in this task.

Table: Table 4: Out-of-domain evaluation results. ^† indicates results from original papers, ^∗ denotes results obtained using the provided online software, N/A indicates methods not applicable or not designed for certain tasks, and - indicates unreported scores in original papers.

### 4.4 Ablation Study

Effect of Training Data Source.
Given that ECGInstruct is compiled from diverse datasets, it is crucial to examine how each dataset contributes to the model’s overall performance. Table 5 presents a comparative analysis of models trained on various dataset combinations. The model trained exclusively on PTB-XL (P) exhibits the lowest performance across all datasets, indicating the limitations of relying on a single data source for effective generalization.
As we progressively incorporate additional datasets into the training set, the model’s performance consistently improves. These results highlight the importance of curating diverse training data, as expanding beyond a single source enhances the model’s capacity to generalize across datasets and tasks.

Table: Table 5: Performance of different training dataset combinations. P: PTB-XL, M: MIMIC-IV-ECG, C: CODE-15%, E: ECG-QA. F1 for PTB-XL Super, CODE-15%, and CPSC; Accuracy for CSN, ECG-QA, G12, and MMMU ECG; Report Scores for PTB-XL Report; Arena Scores for ECG Arena. AVG denotes the average across all metrics.

Effect of Instruction Task.
To understand the individual contribution of each ECG-related task to model performance, we analyze combinations of four instruction tasks. As shown in Table 6, adding more tasks progressively improves performance across multiple benchmarks. Models trained solely on basic feature recognition (F) performed poorly across all metrics, highlighting the limitations of a single-task approach. In contrast, the sequential addition of tasks led to substantial performance gains across multiple benchmarks. The model incorporating all four tasks achieved the highest performance, indicating a more comprehensive understanding of ECG images.

Table: Table 6: Performance of different ECG-related instruction task combinations. F: basic feature recognition, R: heart rhythm analysis, M: morphology and pathology identification, C: clinical report generation. F1 for PTB-XL Super, CODE-15%, and CPSC; Accuracy for CSN, ECG-QA, G12, and MMMU ECG; Report Scores for PTB-XL Report; Arena Scores for ECG Arena. AVG denotes the average across all metrics.

### 4.5 Case Study

We further present some examples from our benchmark, comparing the outputs of our model with GPT-4o for ECG report generation (Appendix Figs. A11-A13) and ECG Arena (Appendix Fig. A14). While GPT-4o is capable of generating reports and answering questions by following instructions, it often produces responses that, although well-structured and seemingly relevant, contain significant inaccuracies in interpretation. In contrast, PULSE consistently provides more accurate responses that align closely with the ground truths. Additionally, we observed that GPT-4o tends to over-rely on its OCR capabilities when textual information (e.g., printed axis labels, numerical values like heart rate or QRS duration) is present in images, leading to superficial reasoning based on text rather than a deep analysis of visual data. As shown in Appendix Fig. A13, GPT-4o identifies a left axis deviation based on the printed QRS axis degree, without analyzing the visual waveform patterns. If such axis information were absent, the model would likely fail to identify the deviation.

### 4.6 Discussion

Figure: Figure 4: Score breakdown of report generation performance.
Refer to caption: x4.png

While the model demonstrates superior performance across various evaluation datasets, it faces notable challenges with more complex and open-ended tasks, such as report generation and multi-turn conversations. To further investigate the model’s performance in report generation, we present the score breakdown in Fig. 4. The model excels in rhythm interpretation but struggles with waveform and diagnosis identification. These results suggest that future efforts should prioritize increasing the dataset’s coverage of waveform and diagnosis-related cases to enhance the model’s ability to detect these abnormalities. Additionally, as diagnosis identification may require more advanced multi-step reasoning, future research could focus on incorporating step-wise instruction tuning data to strengthen the model’s reasoning capabilities.

## 5 Conclusion

In this paper, we study the problem of ECG image interpretation, which is a crucial task in assessing cardiac conditions. We develop a new MLLM, PULSE, fine-tuned on the newly created ECGInstruct dataset with over 1 million samples across a diverse range of ECG-related tasks. Evaluated on the proposed benchmark, ECGBench, our model shows state-of-the-art performance, surpassing both proprietary and open-source MLLMs across multiple in-domain and out-of-domain evaluation datasets. This work demonstrates the potential of using MLLMs for enhancing ECG image analysis and interpretation in clinical applications.

## Appendix A Related Work

#### Domain-specific Models for ECG.

Many domain-specific models have been proposed to enhance automatic ECG diagnosis (Hannun et al., ; Ribeiro et al., ; Hughes et al., ). For example, Ribeiro et al. () applied convolutional neural networks (CNNs) to encode ECG signals for diagnosing 6 types of abnormalities. To reduce dependence on
high-quality labeled data, recent studies (Li et al., ; Liu et al., ; Na et al., ) have further explored self-supervised learning approaches using unlabeled ECG training data. For example, Liu et al. () proposed an ECG representation learning framework by integrating the ECG signals and clinical reports, showing improved performance in zero-shot ECG classification tasks. Despite these successes, most approaches treat ECG data as temporal physiological signals, which could be limiting in certain resource-constrained or remote settings where only printed or digital images are available. Recently, a few methods (Sangha et al., ; ; Khunte et al., ) have been proposed for ECG diagnosis using ECG images. For example, Khunte et al. () developed a diagnostic report generation framework for ECG images, which is built upon a BEiT (Bao et al., ) vision transformer encoder and a GPT-2 (Radford et al., ) decoder. However, their model is only capable of the clinical report generation task, without generalizability to other diverse tasks. In contrast, our study investigates the capabilities of MLLMs for ECG image interpretation. We curate a diverse range of instruction tuning datasets to fine-tune the model, thus improving model generalizability.

#### MLLMs in Healthcare

Recent advancements in MLLMs have shown promising results in various healthcare domains. General medical multimodal models such as LLaVA-Med (Li et al., ), MedPaLM (Singhal et al., ; ), and Med-Gemini (Saab et al., ) have demonstrated capabilities in processing diverse medical data types. Additionally, domain-specific multimodal models have been developed for specialized fields like pathology (Lu et al., ; Xu et al., ) and radiology (Wu et al., ). These models have shown potential in integrating visual and textual information to support clinical decision-making and medical education. However, despite the importance of ECG data in cardiac diagnosis and monitoring, current MLLMs often struggle to process ECG images effectively. This limitation highlights a significant gap in the application of MLLMs to cardiology, where the ability to interpret both visual ECG representations and accompanying clinical information is crucial.

#### Multimodal Instruction Tuning.

Instruction tuning has proven effective in the multimodal domain, particularly in vision-language models like LLaVA (Liu et al., ), MiniGPT-4 (Zhu et al., ) and InstructBLIP (Dai et al., ). These models demonstrate impressive generalizability across various visual understanding and reasoning tasks. While multimodal instruction tuning has been applied to general medical imaging tasks (Li et al., ; Singhal et al., ), its application to ECG images remains largely unexplored. A recent work (Wan et al., ) introduced a targeted instruction tuning framework and fine-tuned existing open-source LLMs for ECG report generation. However, their approach is limited by a single-task instruction dataset focused solely on report generation, potentially constraining its adaptability to other ECG-related tasks. Moreover, their work also treats ECG data as temporal signals, whereas our paper focuses on encoding ECG images with MLLMs, which is more applicable to real scenarios where only printed or digital ECG images are available.

## Appendix B Preliminary on 12-lead ECG

ECG is a vital diagnostic tool that measures the electrical activity of the heart over time, providing insights into both spatial and temporal aspects of cardiac function. Typically, an ECG recording is presented as a 12-lead multivariate time series, where each lead offers a unique perspective on heart activity. The six limb leads (I, II, III, aVR, aVL, and aVF) assess the electrical movements across the arms and legs, giving views from the frontal plane. Simultaneously, the six precordial leads (V1, V2, V3, V4, V5, and V6) monitor the chest, offering horizontal plane views. In this paper, we focus on ECG images that are synthesized from raw signals.

## Appendix C Details of ECG Image Synthesis

We employ the ECG-image-kit (Shivashankara et al., ) framework to synthesize diverse ECG images from raw signal data. This toolkit allows for the generation of ECG images under various conditions by introducing a range of distortions and noises to better simulate real-world clinical data.

Specifically, in addition to generating standard 12-lead ECG images—characterized by black waveforms on a white background, red grid lines, and a 4x3 layout—we introduce a variety of perturbations to the images. These modifications include the addition of wrinkles and creases, simulating the physical wear and tear commonly observed in paper-printed ECGs.
Our image synthesis process includes various augmentation methods to simulate physical distortions, image quality variations, and layout alterations. We introduce wrinkles and creases to mimic wear and tear commonly observed in paper-printed ECGs, and apply random rotations at varying angles to simulate misaligned scans or prints. To account for different acquisition systems and scanning qualities, we vary image resolutions and introduce random background colors, such as slight yellowing to represent aging or poor scanning quality. We also add noise to the images to simulate imperfections in the scanning or printing process. Furthermore, we experiment with different aspect ratios, overall image sizes, and ECG plot positions within the image to reflect the heterogeneity of ECG printouts across different systems and formats. In some cases (with a 0.02 probability), we randomly remove grid lines to account for variations in ECG presentation.

To further enrich the synthetic images, we randomly insert meta-information into the image header to simulate the annotations typically seen in clinical ECG reports. For the PTB-XL dataset, we extract patient demographics (e.g., age, gender) and basic ECG features (e.g., heart rate, axis deviations) from the associated PTB-XL feature annotation dataset, PTB-XL+ (Strodthoff et al., ). This extracted data is used to impute realistic meta-information, which is then randomly printed on the synthesized image. This random insertion of meta-data not only increases the visual variety of the images but also provides additional context, simulating real-world ECG prints that include patient and diagnostic information. To further increase diversity, we adopt alternative lead configurations beyond the standard 4x3 layout, such as 12x1 (single row of leads), 6x2 (two rows of six leads), and other commonly used clinical formats. These variations ensure that our model is exposed to a wide range of ECG presentation styles.

The augmentation process is designed to balance the dataset, with an approximate ratio of 1:1 between augmented and standard ECG images. This balance ensures that the model is exposed to both clean and distorted images, aiding in its generalization to real-world clinical scenarios.

## Appendix D Details of Instruction Tuning Datasets

Table: Table A1: Detailed data statistics of ECGInstruct.

Figure: Figure A1: The Examples of basic feature recognition instructions for finetuning PULSE.
Refer to caption: x5.png

Figure: Figure A2: The Examples of heart rhythm analysis instructions for finetuning PULSE.
Refer to caption: x6.png

Figure: Figure A3: The Examples of morphology and pathological condition identification instructions for finetuning PULSE.
Refer to caption: x7.png

Figure: Figure A4: The Examples of clinical reporting instructions for finetuning PULSE.
Refer to caption: x8.png

## Appendix E Prompts

Figure: Figure A5: The prompt used to synthesize ECG instruction tasks based on clinical reports.
Refer to caption: x9.png

Figure: Figure A6: The prompt used to synthesize ECG multi-turn dialogue as instruction tuning data.
Refer to caption: x10.png

Figure: Figure A7: The prompt used to revise (and translate) original reports.
Refer to caption: x11.png

Figure: Figure A8: The prompt used to score and filter generated instruction data.
Refer to caption: x12.png

Figure: Figure A9: The prompt used to evaluate the generated report.
Refer to caption: x13.png

Figure: Figure A10: The prompt used to evaluate the ECG Arena.
Refer to caption: x14.png

## Appendix F Details of Evaluation Metrics

Abnormality Detection. we utilize multi-label classification metrics, including Macro AUC, Macro F1, and Hamming Loss, to evaluate the datasets PTB-XL Super, CODE-15%, and CPSC 2018, where multiple correct labels may exist. For the ECG-QA, CSN, and G12EC datasets, we adopt accuracy as the evaluation metric.

Report Generation. Rather than relying on traditional text generation metrics, we leverage strong LLMs as evaluators, following the approach of Zheng et al. (). This method provides a more nuanced evaluation by focusing on key aspects of the reports. Specifically, we use GPT-4o to compare the model-generated reports against those written by cardiologists. We introduce a “Report Perfect Score”, which is based on three critical components of a generated report: (1) Rhythms (0 to 10 points), (2) Waveform Morphology (0 to 10 points), and (3) Diagnosis (0 to 10 points). The final score is the average of these three components, scaled to a maximum of 100 points. The prompt used to query GPT-4o for evaluating the report score is provided in Appendix Fig. A9.

MMMU ECG. We adopt accuracy as the primary metric. We have designed systematic, rule-based evaluation pipelines to ensure robust and consistent scoring. To mitigate the potential influence of any intermediate generations (e.g., reasoning steps) in long responses, we employ robust regular expressions and develop response-processing workflows. These are used to extract answer options from the long responses for accurate answer matching. In cases where no valid answer can be extracted from the model’s response, we perform random selection to assign a score.

ECG Arena. We also employ a strong judge model, GPT-4o, to assess model performance by comparing generated responses with ground truth answers. The evaluation considers three perspectives, each scored on a scale of 0-10: Accuracy (how closely the model’s response matches the ground truth), Completeness (whether the model provides a comprehensive answer covering all aspects of ECG interpretation), and Instruction Adherence (how well the model follows the specific instructions in the question). We calculate the final score by averaging these three aspects and scaling to a maximum of 100 points. The specific prompt used for GPT-4 evaluation is provided in Appendix Fig. A10.

## Appendix G Case Study

Figure: Figure A11: Comparison of model outputs on ECG report generation task (Example 1). Blue indicates correct information, while red highlights errors. Our model’s output fully aligns with the ground truth, with a report score of 10. In comparison, GPT-4’s report, though structurally sound, contains notable inaccuracies despite its initial appearance of relevance.
Refer to caption: x15.png

Figure: Figure A12: Comparison of model outputs on ECG report generation task (Example 2). Blue indicates correct information, while red highlights errors. Our model’s output mostly aligns with the ground truth, achieving a report score of 83.3. In comparison, GPT-4’s output correctly identifies only the ECG rhythm, omitting most other key details.
Refer to caption: x16.png

Figure: Figure A13: Comparison of model outputs on ECG report generation task (Example 3). Blue indicates correct information, while red highlights errors. Our model’s output mostly aligns with the ground truth report, achieving a report score of 73. In comparison, GPT-4’s output partially aligns with the ground truth report.
Refer to caption: x17.png

Figure: Figure A14: Comparison of model outputs on ECG Arena (Example 1). Blue indicates correct information, while red highlights errors. Given the challenging nature of this task, our model’s output partially aligns with the ground truth, and GPT-4o’s output largely deviates from the reference.
Refer to caption: x18.png

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: Please provide technical details.
For a fairer comparison, it would be preferable to include fine-tuning results against other LLMs. 
Additionally, human accuracy should be evaluated, at least partially, for the ECGBench tasks. 
It is also recommended to assess performance on external validation tasks beyond the ECGBench dataset.
soundness: 3
strengths: The study’s strengths include the development of a large-scale dataset and the introduction of extensive benchmark tasks, both of which contribute positively to the field.
summary: This study presents the creation of a fine-tuning dataset and benchmark for ECG in multi-language language models (MLLM), which appears to hold value for advancing medical AI. The dataset is synthetically generated with various distortion noises, and Llama 3 was used for ECGInstruct and quality checking; however, the accuracy of these implementations has not been assessed, raising concerns about potential LLM bias. Therefore, evaluation on independent downstream task datasets would be essential.
weaknesses: The study lacks a strong technical novelty and technical details.
For a fairer comparison, it would be preferable to include fine-tuning results against other LLMs. 
Additionally, human accuracy should be evaluated, at least partially, for the ECGBench tasks. 
It is also recommended to assess performance on external validation tasks beyond the ECGBench dataset.

---

confidence: 3
contribution: 3
initial_rating: 8
presentation: 3
questions: According to table 4, the performance gain in arena score which reflects the model’s instruction-following ability is limited compared with Claude 3.5 Sonnet in arena score. The authors could include more discussion about the results to help improve multistep reasoning capabilities in future research.
soundness: 3
strengths: This work contributes a comprehensive ECG image instruction tuning dataset from diverse data sources, which facilitates the advancement of MLLMs in understanding ECG images.
The proposed PULSE model surpassed proprietary and open-source MLLMs by a large range in diverse tasks related to ECG image comprehension. The model, data and code have been released.
summary: This paper introduces PULSE, a multimodal large language model (MLLM) tailored for interpreting electrocardiogram (ECG) images, addressing the limitations of existing methods which often rely on raw physiological signals and lack generalizability. The authors contribute a new dataset, ECGInstruct, comprising over one million ECG image-text samples to fine-tune PULSE, and ECGBench. Experiments demonstrate that PULSE outperforms both proprietary and open-source MLLMs, with an average accuracy improvement of 15% to 30%.
weaknesses: As stated in the discussion, the dataset contains few multistep instructions, which could undermine the multistep reasoning capability of the PULSE model and limits the report generation performance.

---

confidence: 4
contribution: 1
initial_rating: 3
presentation: 3
questions: - Are there many use cases, where ECGs are not available in electronic form, but where scanners and AI models are available and could be integrated?
- Even if only printouts are available, is directly interpreting them as images the best solution? Wouldn’t the more reliable solution be to convert these images into electronic ECGs first (the problem seems relatively simply due to the high contrast of the plots and often clearly visible reference lines) and the use established ECG models?
- Motivation lacks reasoning: Why would the authors transform readily available ECG recordings, e.g. 800k samples from MIMIC-ECG, into images before analysing them? The majority of the transformed image represents background, i.e. noise. Is the patch projector of the imaging encoder (ViT) capable of learning a meaningful signal from the data?
- It would have been interesting to see how much using the ECG-images instead of the raw ECGs limits the performance.
- An ablation study using an ECG encoder instead of the image model would be interesting
- An ablation study using the image encoder directly on ECG images, without the LLMs would be interesting
- To provide robustness guarantees, have the authors evaluated their methods as well as the baselines across multiple seeds set during fine-tuning?
soundness: 2
strengths: The paper's strengths include:

- many SOTA MLLMs as baselines
- Evaluation on a wide range of tasks
- Preparation of a benchmark that could be used in future works. This may be relevant for the field.
- Preparation of an instruction-tuning dataset that could be used in future works. Again, this may be relevant for the field.
summary: The paper proposes a fine-tuned multi-modal LLM tailored for ECG image interpretation. Additionally, the paper introduces a large ECG image instruction tuning dataset with 1 million examples, covering a wide range of ECG-related tasks from diverse data sources. A new evaluation benchmark for ECG image interpretation tasks is also proposed.
weaknesses: The weaknesses include:

- Motivation: while there may be cases where only printouts of ECGs are available, the need for AI models on ECG printouts is still questionable.
- Related works: Current multimodally trained ECG models are neither discussed nor compared against, e.g.:
Radhakrishnan et al. (2023), "Cross-modal autoencoder framework learns holistic representations of cardiovascular state", https://www.nature.com/articles/s41467-023-38125-0, Turgut et al. (2023), "Unlocking the Diagnostic Potential of ECG through Knowledge Transfer from Cardiac MRI", https://arxiv.org/pdf/2308.05764
- Current foundational time series models are neither discussed nor compared against, e.g.: Yang et al. (2023), "BIOT: Biosignal Transformer for Cross-data Learning in the Wild", https://proceedings.neurips.cc/paper_files/paper/2023/hash/f6b30f3e2dd9cb53bbf2024402d02295-Abstract-Conference.html, - It would have been interesting to see how much using the ECG-images instead of the raw ECGs limits the performance.
- An ablation study using an ECG encoder instead of the image model would be interesting
- An ablation study using the image encoder directly on ECG images, without the LLMs would be interesting
- To provide robustness guarantees, have the authors evaluated their methods as well as the baselines across multiple seeds set during fine-tuning?Goswami et al. (2024), "MOMENT: A Family of Open Time-series Foundation Models", https://arxiv.org/pdf/2402.03885
- Statements such as "limitations of traditional ECG models" (ll 64-65) are made without elaborating and providing references.
- Experiments / Evaluation: For fair comparison, Table 3 and 4 should include the ECG baselines mentioned above. This would highlight the differences between imaging and time series modality with respect to downstream performance.
- Furthermore, domain-specific methods in Table 3 and 4 have not been tuned for the reported downstream tasks. The authors make the effort to tune multiple MLLMs on the applications under investigation, however, they only report scores from the original paper for the domain-specific methods.
- In Table 4, MERL achieves best performance on the CPSC 2018 task, however, the authors mistakenly highlight their own method PULSE as the best performing one.
- In Table 3 and 4, the authors report that certain methods are not applicable or not designed for certain tasks, which is why they opt to not report scores in those cases. However, for the open-source MLLMs that are not designed for ECG image analysis either, the authors tune the models and report performance scores. This clearly is an unfair comparison, further highlighting the need for tuning the domain-specific methods and reporting their performance.
- All LLM-based baselines have not been trained on ECGs / ECG-images, while ECG baselines are not able to handle all tasks. So for some of the tasks (the text-based ones) no meaningful baseline is provided. While such baselines may not be available, the quality of the model could still be compared with strong baselines by splitting it into subtasks, i.e. use established standard models to extract relevant features from a given ECG (heart rate, irregularities, abnormalities, …). Then include the extracted features in a prompt and ask a general or medical LLM to provide an answer. This could handle most of the tasks and thus provide a meaningful baseline.
- Limited novelty: This seems to be a very applied paper without any novelty in the method (note that the provided instruction tuning and evaluation datasets are still relevant for the field)

---

confidence: 5
contribution: 2
initial_rating: 5
presentation: 3
questions: See weaknesses above.
soundness: 3
strengths: - The first work to treat ECG signals as visual signals rather than time series.
- Builds the first benchmark for ECG instruction tuning with image input.

To the best of the reviewer's knowledge, this is the first work converting ECG signals into visual input, allowing ECG analysis to benefit from the advancements in the Visual Language Model (VLM) community. Additionally, the authors establish a comprehensive benchmark for evaluating ECG image instruction tuning, opening a new domain for ECG analysis.
summary: This work proposes an impressive framework for instruction-tuning ECG images, evaluates it across multiple datasets, and introduces the first benchmark for ECG image instruction tuning.
weaknesses: Even though this work is well-executed and evaluated on multiple datasets, it still has some weaknesses:

- **Reliance on Accuracy and LLM Scoring**: The evaluation metrics are based solely on accuracy or LLM scoring. Incorporating human evaluation could enhance benchmark quality, as current LLMs are not specialized for ECG-related text understanding. I suggest using human evaluation for all open-ended tasks.

- **ViT Model Limitations**: The Vision Transformer (ViT) model is frozen, and while LLAVA uses ViT, it is not specifically tuned for ECG image features. This may degrade performance on ECG-specific tasks.

- **Translation Issues in Report Generation**: For report generation, the PTB-XL dataset originally includes German, Danish, and other European languages, not English. Translating these reports into English could lead to inaccuracies or loss of meaning.

- **Evaluation Metrics in Report Generation**: In the report generation task, only GPT is used as the LLM judge, without any lexical or clinical efficacy metrics. Clinical efficacy metrics are commonly used in other report generation tasks, such as in [1].

- **Relevance of Abnormality Detection as an LLM Task**: Is abnormality detection a necessary task for an LLM? According to Section 3.2, this task appears similar to classification. Additionally, LLMs can generate synonyms for disease names, raising the question of how to handle variations in terminology.


[1]Tanida, Tim, et al. "Interactive and explainable region-guided radiology report generation." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.