DOI: 10.5281/zenodo.15893350
title: MEETI: A Multimodal ECG Dataset from MIMIC-IV-ECG with Signals, Images, Features and Interpretations

description: Announcements





Publication Status: The work related to this dataset has been accepted for publication in Scientific Data. Updated citation information will be provided shortly.




Dataset Content: The dataset consists of four modalities: (1) raw signals, (2) plotted images, (3) per-beat parameters, and (4) interpretation text. The current processing status is as follows:





Raw signals: Completed




Per-beat parameters: Completed




Interpretation text: Completed




Plotted images: Generating...






Note on Updates: Due to the large volume of image data (approx. 800,000 images), we will release a multimodal subset containing nearly 10,000 samples within one week. The full dataset update will be completed within the next three weeks.


Key Events:

          Multimodal data of approximately 10,000 samples     Completed

          Complete MEETI dataset     Updating  (before 02/28/2026)



 

Abstract 

Electrocardiograms (ECGs) are essential for diagnosing arrhythmias, myocardial ischemia, and conduction disorders. While machine learning has achieved expert-level performance in ECG interpretation, the development of clinically deployable multimodal aI systems is limited by the lack of public datasets that integrate raw signals, diagnostic images, and interpretation text. Most existing ECG datasets are single-modality or include, at most, signal-text pairs, restricting real-world applicability. To address this gap, we present MEETI (MIMIC-IV-Ext ECG-Text-Image), the first large-scale dataset that synchronizes raw ECG waveforms, high-resolution plotted images, and detailed textual interpretations generated by large language models. MEEtI also includes beat-level quantitative parameters extracted from each lead, enabling fine-grained analysis and improving model interpretability. Built on the MIMIC-IV-ECG database of over 800,000 recordings, MEETI aligns each record across four components using unique identifiers: (1) raw signals, (2) plotted images, (3) per-beat parameters, and (4) interpretation text. This structure enables multimodal transformer learning and supports explainable, integrated analysis. MEEtI provides a robust foundation and benchmark for next-generation cardiovascular artificial intelligence research.

Background 

Cardiovascular diseases (CVDs) are the leading cause of mortality worldwide [1], contributing to over 17 million deaths each year. Electrocardiography (ECG) remains the primary noninvasive modality for assessing cardiac electrophysiology. It provides temporal waveform information that is critical for diagnosing arrhythmias, myocardial ischemia, and conduction abnormalities [2]. However, manually interpreting large volumes of ECG recordings is time-consuming and prone to inter-observer variability. Recently, machine learning and AI approaches have emerged as powerful tools for ECG analysis [3]. They can achieve expert-level performance in waveform delineation, arrhythmia detection, and risk stratification. These advances underscore the urgent need for comprehensive, multimodal ECG datasets. Such datasets are essential to ensure reproducible research and to accelerate clinical translation. Over the past few years, multimodal large language models have demonstrated remarkable performance in interpreting complex medical imaging data [4]. They can seamlessly integrate radiographic, histopathological, and clinical text inputs to support diagnostic and prognostic workflows. These advanced architectures use transformer-based encoders to fuse pixel-level features with semantic context. As a result, tasks such as lesion detection, disease grading, and report generation can be performed at near-human accuracy. Despite these successes, current large language models remain incapable of directly processing and understanding electrocardiographic waveforms. Cardiac signals have unique temporal dynamics and amplitude variations that differ fundamentally from spatial image patterns. Bridging this gap requires specialized datasets and model designs that jointly represent ECG signals alongside complementary textual and visual information.

Given that ECG data inherently combine visual patterns, temporal dynamics, and semantic information, their multimodal nature has long been recognized in both clinical and computational analyses. Traditional ECG pipelines rely on handcrafted feature extraction (e.g., QRS duration, ST-segment deviations) to support diagnostic decisions [5]. In contrast, recent deep learning approaches achieve end-to-end performance gains but are often criticized as "black boxes" lacking explicit interpretability and semantic features [6]. These limitations underscore the need to model ECGs through an integrated, multimodal lens. As a result, we need a unified “ECG multimodal” approach. Training multimodal language models to understand ECG data requires genuine “ECG multimodal” examples—paired waveform images, raw signals, and textual annotations. However, existing public ECG datasets only include signal–text pairs and leave out image data. This gap prevents a systematic study of how to combine visual and temporal cardiac representations in transformer-based models. The lack of integrated datasets is therefore a bottleneck for ECG multimodal research. It motivates the creation of resources that encode images, signals, and semantics together for end-to-end model training.

To address these limitations, we introduce MEETI, a novel dataset designed to improve large language models’ understanding of electrocardiograms. MEETI is built on the MIMIC-IV-ECG dataset [7], one of the largest and most representative public 12-lead clinical ECG collections available. It covers various cardiac conditions and signal qualities that reflect real-world clinical settings. MEETI is the first ECG multimodal dataset to include synchronized ECG images, structured diagnostic text, and expert-annotated signal reports. By linking ECG waveforms, image features, and domain-specific knowledge representations, MEETI provides a unified triadic framework. This framework offers a standardized input format for training multimodal large language models and promotes explainable AI by enabling clinically interpretable reasoning rather than “black-box” classification. Integrating visual, temporal, and semantic modalities allows for cross-modal alignment and supports the development of models capable of multi-perspective ECG analysis. In addition, MEETI encourages a shift from traditional end-to-end diagnostic pipelines toward inference-based reasoning. Ultimately, this resource enhances the ability of multimodal large language models to generate meaningful, context-aware interpretations of cardiac health.

Methods

ECG signals and reports

As part of routine clinical care at the Beth Israel Deaconess Medical Center (BIDMC), diagnostic 12-lead ECGs were collected using standard ECG machines and subsequently archived in the MIMIC-IV-ECG [7]. This dataset comprises approximately 800,000 10-second, 12-lead ECG recordings sampled at 500 Hz, matched to nearly 160,000 unique patients included in the broader MIMIC-IV Clinical Database. In addition to the raw waveform data, machine-generated measurements and cardiologist-interpreted reports are also available for a substantial proportion of ECGs. Most ECG waveforms get read by a cardiologist, and an associated report is generated from the reading. All records have been de-identified by the HIPAA Safe Harbor requirements to ensure patient privacy.

ECG feature extraction

To extract a comprehensive set of quantitative descriptors from the MEETI ECG waveforms, we integrated the open‑source FeatureDB toolkit [5] (https://github.com/PKUDigitalHealth/FeatureDB) into our preprocessing pipeline. FeatureDB employs a multi‐stage workflow beginning with adaptive peak detection to localize the P wave, QRS complex, and T wave across each lead. Following initial peak localization, discrete wavelet transforms are applied at multiple scales to refine the onset and offset boundaries of these waveform components, thereby ensuring precise measurement of key temporal intervals (PR interval, QRS duration, QT interval, and QTc). We applied this pipeline to all de‐identified 12‑lead ECG records after verifying signal integrity (complete lead set, consistent sampling rate, and absence of major artifacts). Extracted features were stored in a structured relational database, with each record linked to its raw WFDB waveform files and corresponding expert reports via unique study identifiers. This systematic application of FeatureDB not only facilitates scalable, reproducible extraction of clinically meaningful ECG metrics but also lays the groundwork for downstream multimodal integration with image and text modalities, ultimately supporting the development of interpretable AI models in cardiac electrophysiology.

ECG plot

To generate high-fidelity ECG images, we rendered each preprocessed 12‑lead waveform into a standardized graphical format using the open‑source ecg_plot Python library (https://github.com/PKUDigitalHealth/ecg_plot). For each record, we configured the plotting parameters to emulate clinical paper recordings: a paper speed of 25 mm/s and an amplitude scaling of 10 mm/mV. Under these settings, each large grid block corresponds to 0.5 mV in voltage and 0.2 s in time, thereby ensuring precise spatial encoding of waveform morphology. Leads were arranged in the conventional 3×4, 6×2, or 12×1 (I, II, III; aVR, aVL, aVF; V1–V6) with appropriate inter‑panel spacing and grid visibility. Images were exported at 100 dpi resolution to facilitate downstream vision‑based model training and to maintain pixel-level detail of low-amplitude features. Each image file was named using the study identifier, then stored alongside the corresponding waveform and text report for seamless multimodal alignment. This procedural standardization yields a uniform image corpus suitable for transformer-based and convolutional model development in ECG interpretation.

ECG interpretation generation

Human-written ECG reports are typically brief and often lack the detail needed for comprehensive interpretation. To produce high-granularity ECG interpretations that explicitly link diagnoses to measurable parameters such as QRS durations or PR intervals, we develop role-based prompts that, combined with extracted ECG features, allow GPT-4o to effectively utilize its latent medical knowledge for generating clinically detailed and feature-grounded interpretations. For each ECG record, the extracted features are structured into sequential data. For example, an ECG signal containing ten visible heartbeats allows us to construct a QRS duration sequence as ([QRS_1, QRS_2, ..., QRS_{10}]), where each element precisely represents the QRS duration for an individual heartbeat. These sequences offer fine-grained descriptions of ECG physiology, enabling assessment of cardiac function at the individual heartbeat level. The role-based prompts then guide GPT-4o with cardiology-specific instructions to analyze various aspects of the ECG features, conditioned on the ground-truth report. For example, the prompt may instruct GPT-4o to examine P wave amplitude and duration in Lead II for signs of atrial enlargement. These instructions also reflect the logical structure of clinical diagnostic workflows. Through this approach, we can generate high-granularity ECG interpretations at scale without relying on manual annotations.

Data Description

MEETI consists of four complementary components: raw ECG signals and their corresponding clinical reports, plotted ECG images, beat-level ECG measurements, and textual interpretations generated by large language models (LLMs). Except for a small number of samples that the algorithm could not process, MEETI covers the full MIMIC-IV dataset, up to 784680 records from 160597 patients. Each component is linked by a unique record identifier so that users can cross-reference signals, images, features, and interpretations. Our goal is to lower the barrier to entry for multimodal ECG research. By making all four data types available in one cohesive resource, we hope to accelerate developments in ECG-based diagnostics, prognostics, and educational tools. Researchers can seamlessly integrate waveform analysis, image-based deep learning, feature engineering, and natural language processing within a single dataset. MEETI thus represents a comprehensive, publicly accessible resource for advancing AI in cardiology.



Raw ECG data and report: The raw ECG signals and accompanying text reports are included directly from MIMIC-IV-ECG [7]. These original signals cover a wide range of patient demographics, clinical conditions, and recording settings. Researchers can access the same high-resolution waveform data that clinicians used at Beth Israel Deaconess Medical Center. By depositing MEETI on zenodo, we ensure long-term accessibility and enable others to reproduce our results without additional data collection efforts.

Plotted ECG image: MEETI provides a set of plotted ECG images. Each image arranges twelve leads in a 12×1 grid, replicating the standard clinical layout. We generated these plots using the matplotlib library. The plotting scripts apply consistent scaling and labeling conventions to preserve the clinical context. Clinicians and machine learning practitioners alike can use these images for visual analysis, teaching, or training convolutional neural networks. Since plotted images are derived directly from the same high-fidelity signals, they faithfully represent the original ECG morphology.

ECG parameter: MEETI also includes quantitative ECG features for every beat in each lead [5]. These parameters were automatically extracted using the FeatureDB toolkit. For each lead, FeatureDB identifies fiducial points (P-wave, QRS complex, T-wave) and computes intervals (PR, QRS duration, QT/QTc) as well as amplitudes (R- and S-wave heights in leads V1 and V5, for example). By capturing beat-by-beat variability, these detailed metrics support both statistical analyses and deep learning approaches. Researchers interested in heart rate variability, conduction delays, or repolarization abnormalities will find this dataset especially useful. Providing per-beat features from all twelve leads means that MEETI can facilitate studies of subtle waveform changes that might be missed when relying on summary statistics alone.

Interpretation of the LLM: MEETI offers natural language interpretations generated by an LLM. We used GPT-4o to produce these interpretations based on the combined input of the original textual reports and the extracted ECG features. For each recording, the LLM output highlights significant findings, such as rhythm abnormalities or evidence of ischemia. By comparing LLM-generated text with clinician-authored reports, users can evaluate the model’s diagnostic suggestions. These interpretations serve as a benchmark for future work on AI-driven ECG reporting, and they demonstrate how multimodal inputs (text plus structured parameters) can inform more accurate, context-aware explanations.


Each subject’s data is organized into a dedicated folder named by their unique subject ID, which corresponds directly to the IDs used in the MIMIC-IV database. Within each folder, plotted ECG images are saved as individual PNG files, with filenames that reflect lead arrangement and recording timestamps. Beat-by-beat parameters—such as PR interval, QRS duration, QT/QTc values, and fiducial point amplitudes—are stored in the MAT file. This file includes detailed metrics for every beat across all twelve leads, making it easy to perform quantitative analyses without further preprocessing. The LLM-generated textual interpretations from GPT-4o are also provided in the MAT file. Those interpretations link directly to the corresponding ECG feature sets, allowing users to compare AI-driven commentary with numerical results. By grouping all PNG and MAT files under a single subject-specific directory, MEETI ensures that researchers can quickly locate ECG images, per-beat ECG features, and LLM-based textual interpretations for each individual. This streamlined structure supports seamless multimodal analysis.

The ECGs are grouped into subdirectories based on subject_id. Each record path follows the pattern: files/pNNNN/pXXXXXXXX/sZZZZZZZZ/ZZZZZZZZ, where:



NNNN is the first four characters of the subject_id,

XXXXXXXX is the subject_id,

ZZZZZZZZ is the file_name


An example of the file structure is as follows:

MEETI
|——— p1000
   |——— p10000032
      |——— s40689238
         |——— 40689238.mat......
|——— p1571
   |——— p15718729
      |——— s40000369
         |——— 40000369.mat
         |——— 40000369.png      |——— s40461152         |——— 40461152.mat      |——— s44659160         |——— 44659160.mat      |——— s48122836         |——— 48122836.mat

Above we find two subjects p10000032 (under the p1000 group level directory) and p15718729 (under the p1571group level directory). For subject p10000032 we find one study: s40689238. For p15718729 we find four studies: s40000369, s40461152, s44659160, s48122836. The study identifiers are completely random, and their order has no implications for the chronological order of the actual studies. Each study has a like-named .png and .mat file.

The record_list.csv file contains the file name and path for each record. It also provides the corresponding subject_id and file_name. The subject_id can be used to link a subject from MEETI to the other modules in the MIMIC-IV-ECG Database [7].

Usage Notes

This module gives MEETI users an additional, potentially valuable source of information for research involving the electrocardiogram large language model.

The dataset does have some limitations. Because of image size constraints, it includes about 10,000 electrocardiogram images. Researchers who need more ECG image data can use MIMIC-IV-ECG [7] as a foundation to generate additional samples from the GitHub repository (https://github.com/PKUDigitalHealth/ecg_plot).

Below is a basic script for reading the downloaded records from this project:



from PIL import Imagefrom scipy.io import loadmatloaddata = loadmat('./p1571/p15718729/s40000369/40000369.mat')print(loaddata.keys())print(loaddata['id'])print(loaddata['LLM_Interpretation'])print(loaddata['report'])img = Image.open('./p1571/p15718729/s40000369/40000369.png')print(img.size)

Our recent ECG MLLM work, GEM, which also used MEETI, can be found at [8].