SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Jungwoo Oh $^{1}$ , Gyubok Lee $^{1}$ , Seongsu Bae $^{1}$ , Joon-myoung Kwon $^{2}$ , Edward Choi $^{1}$   
KAIST, Daejeon $^{1}$  Medical AI Inc., Seoul $^{2}$   
{ojw0123, gyubok.lee, seongsu, edwardchoi}@kaist.ac.kr $^{1}$   
cto@medicalai.com $^{2}$
Question answering (QA) in the field of healthcare has received much attention due to significant advancements in natural language processing. However, existing healthcare QA datasets primarily focus on medical images, clinical notes, or structured electronic health record tables. This leaves the vast potential of combining electrocardiogram (ECG) data with these systems largely untapped. To address this gap, we present ECG-QA, the first QA dataset specifically designed for ECG analysis. The dataset comprises a total of 70 question templates that cover a wide range of clinically relevant ECG topics, each validated by an ECG expert to ensure their clinical utility. As a result, our dataset includes diverse ECG interpretation questions, including those that require a comparative analysis of two different ECGs. In addition, we have conducted numerous experiments to provide valuable insights for future research directions. We believe that ECG-QA will serve as a valuable resource for the development of intelligent QA systems capable of assisting clinicians in ECG interpretations.
In recent years, significant advancements in natural language processing have revolutionized the field of question answering (QA) in a wide range of domains. Previous works have demonstrated the great potential of QA systems in various domains, where they have been combined with different modalities such as images [1, 32, 15, 8] or tables with images [24, 13]. Concurrently, QA systems have also been explored in the healthcare domain, including visual QA with chest X-ray [11, 14], clinical-note-based QA [20], and QA over structured electronic health record (EHR) data [28, 12]. These pioneering efforts have successfully bridged the gap between general-domain QA and the medical field, unlocking new possibilities to improve healthcare outcomes and enhance medical decision-making processes.
Despite this remarkable progress, there is a noticeable absence of datasets that combine electrocardiogram (ECG) data with question answering. As a fundamental diagnostic tool in cardiology, ECG provides critical insights into the electrical activity of the heart and plays an important role in detecting various cardiac conditions [3, 22, 33]. Consequently, integrating ECG data with QA systems holds tremendous potential to improve the interpretation of cardiac data, leading to more accurate diagnoses and personalized treatment plans.
To this end, we present ECG-  $\mathrm{QA}^1$ , a novel QA dataset that incorporates ECG data for question answering tasks. To the best of our knowledge, ECG-QA is the first dataset that combines QA and ECG, opening up new avenues for integrating multi-modal machine learning with cardiac healthcare.
The main contributions of this work are threefold:
<sup>1</sup>The dataset is available at https://github.com/Jwoo5/ecg-qa, licensed under CC-BY-4.0 license.
(a) Questions to a single ECG
(b) Comparison questions between two different ECGs
Figure 1: Sample question-answer pairs in ECG-QA. (a) Questions to a single ECG with various types of attributes. (b) Comparison questions between two different ECGs. Refer to Section 3.1.1 for more details about each attribute type.
- We propose the ECG-QA dataset, a diverse collection of questions focused on ECG interpretation and analysis (See Figure 1). This dataset introduces the novel concept of incorporating question answering into the realm of ECG analysis, making it a valuable resource for developing and evaluating QA systems in the context of cardiology.
- To cover more complex yet clinically critical questions, we include questions that require comparative analysis of two ECGs (See Figure 1 (b)). This inclusion brings a new degree of complexity, as addressing these comparison questions extends beyond the conventional scope of ECG analysis using machine learning. By incorporating these types of questions, we not only address the real-world needs of medical professionals but also broaden the potential applications of machine learning in ECG analysis.
- We provide a benchmark for QA models, including recent large language models (LLMs), on the ECG-QA dataset, promoting further research and encouraging the development of novel methods to leverage ECG signals for question answering tasks. We believe that ECG-QA will serve as a valuable resource in advancing machine learning applications in cardiology and improving medical decision-making processes.
Medical QA datasets QA systems have been extensively explored in the healthcare domain, catering to the specific needs and challenges of medical data. However, most existing medical QA datasets are primarily based on clinical texts, medical images, or structured EHR tables. For example, Pampari et al. [20] proposed the emrQA dataset, consisting of question-answer pairs derived from unstructured clinical notes. In addition, Kovaleva et al. [11] and Liu et al. [14] proposed datasets for visual QA using X-ray images, aiming to investigate intelligent interactive systems for radiology. Meanwhile, in the field of QA over structured EHR data, Wang et al. [28] and Lee et al. [12] have developed datasets called MIMICSQL and EHRSQL respectively, which consist of questions and their corresponding SQL queries. While these healthcare QA systems demonstrate the potential of leveraging medical data with QA to improve healthcare outcomes, there is currently no dedicated QA dataset specifically designed for ECG data despite its widespread use in diagnosing cardiovascular conditions and monitoring patients' heart health.
Electrocardiogram Previous studies in the field of ECG have predominantly focused on using ECG data for diagnostic purposes such as identifying various heart diseases. For instance, Nejedly et al. [18] proposed an ensemble of residual networks with attention modules to classify cardiac diseases using ECGs, which won first place in the PhysioNet/Computing in Cardiology Challenge 2021 [21]. At the same challenge, Han et al. [6] achieved second place by utilizing SE-WRN [31], which is a
Table 1: Sample template questions for different question & attribute types in ECG-QA.
Question type	Attribute type	Example template question
Single-Verify	SCP Code	Does this ECG show symptoms of non-specific ST changes?
Noise	Does this ECG show baseline drift in lead I?
Stage of infarction	Does this ECG show early stage of myocardial infarction?
Extra systole	Does this ECG show ventricular extrasystoles?
Heart axis	Does this ECG show left axis deviation?
Numeric feature	Does the RR interval of this ECG fall within the normal range?
Single-Choose	SCP Code	Which symptom does this ECG show, conduction disturbance or hypertrophy?
Noise	Which noise does this ECG show, baseline drift or static noise?
Stage of infarction	Which stage of infarction is this ECG at, early stage of myocardial infarction or late stage of myocardial infarction?
Extra systole	Which kind of extra systoles does this ECG show, ventricular extrasystoles or supraventricular extrasystoles?
Heart axis	Which cardiac axis does this ECG show, left axis deviation or right axis deviation?
Numeric feature	Which range does the RR interval of this ECG fall in, below the normal range or within the normal range?
Single-Query	SCP Code	What form-related symptoms does this ECG show?
Noise	What kind of noises does this ECG show in lead I?
Stage of infarction	What stage of infarction is this ECG at?
Extra systole	What kind of extra systoles does this ECG show?
Heart axis	What direction is this ECG deviated to?
Numeric feature	What range does the RR interval of this ECG fall in?
Comparison-Consecutive-Verify	SCP Code	Compared to the previous tracing, has left ventricular hypertrophy been resolved in the recent tracing?
Numeric feature	Compared to the previous tracing, has the PR interval of the recent tracing become normal?
Comparison-Consecutive-Query	SCP Code	What symptoms have been resolved in the recent tracing as compared to the previous one?
Numeric feature	What numeric features of the recent tracing now have become normal compared to the previous one?
Comparison-irrelevant-Verify	SCP Code	Compared to the first ECG, has atrial fibrillation been newly detected in the second ECG?
Numeric feature	Compared to the first ECG, has the P duration of the second ECG changed to an abnormal value?
Comparison-Irrelevant-Query	SCP Code	What symptoms still remain in the second ECG as compared to the first ECG?
Numeric feature	What numeric features of the second ECG are now considered abnormal values as compared to the first ECG?
combination of wide residual network [30] and squeeze-and-excitation modules [7]. Furthermore, several works [4, 10, 19] studied self-supervised learning with ECGs to improve performances on the cardiac arrhythmia classification task. These works concentrate on classifying diagnoses based on a single ECG, and do not consider the significance of comparing two ECGs, despite its importance in clinical contexts. For example, by detecting resolved symptoms after some treatments, the medical practitioners can assess the effectiveness of the treatments and evaluate the progress of the patient's condition. To reflect this clinical reality, we have included questions that involve the comparison of two ECGs within the ECG-QA dataset, making our dataset unique and valuable.
We constructed the ECG-QA dataset upon the PTB-XL dataset  $[26]^2$ , which offers comprehensive metadata regarding ECGs annotated by expert cardiologists. This metadata covers a wide range of information including ECG reports, diagnostic statements, diagnosis likelihoods, and signal-specific properties. To ensure the high quality of our dataset, we performed additional filtering on the original
This dataset is licensed under CC-BY-4.0 license.
Figure 2: Visualization of the ECG-QA sample generation pipeline. The numbers in the sampling stage (d) stand for the ECG IDs in the PTB-XL dataset. In the sampling process, we also convert the template questions into pre-defined paraphrases for each sample.
PTB-XL dataset. Specifically, we selected ECGs that were marked with a validated_by_human tag set to True, which indicates the validation by a human cardiologist, and excluded ECGs that had empty reports. As a result, the ECG-QA dataset was constructed using 16,054 samples of 10-second ECGs from the PTB-XL dataset. In addition, we split the samples into training and test sets according to a 8:2 ratio based on their patient IDs before generating QA samples to prevent the overlapping of ECGs between training and test sets. We again split the training samples into training and validation sets by a 9:1 ratio, yielding 7.2:0.8:2.0 training-validation-test distribution.
3.1 Question template
To generate QA samples, we start by creating the question templates to collect questions, answers, and their corresponding ECGs. Because the questions are fully derived from these templates, it is important to define templates that are not only diverse but also clinically meaningful. To achieve this goal, we extracted the relevant attributes from the PTB-XL metadata to determine the content of the questions (i.e., attribute types) and categorized the questions into several question types. Then, we combined these types to construct the template questions, and additionally generated template paraphrases to add lexical diversity to our dataset. As a result, we defined a total of 70 templates as shown in Supplementary A.2, and we also provide the example questions derived from these templates in Table 1. All the processes of designing question templates have been validated by a board-certified medical expert from the Department of Critical Care and Emergency Medicine in terms of clinical utility. The detailed processes of each step are described in the following subsections, as well as visualized in Figure 2 (a) and (b).
3.1.1 Attribute type extraction
SCP code The PTB-XL dataset provides SCP codes for each ECG sample, consisting of 71 different ECG symptoms that adhere to the SCP-ECG v0.4 that preceded the current SCP-ECG standard [23]. These attributes are composed of form-related (e.g., inverted  $T$ -waves), rhythm-related (e.g., sinus arrhythmia), and diagnostic symptoms (e.g., non-specific ischemic) along with additional 5 superclasses for diagnostic labels. Given that detecting cardiac symptoms is a primary objective in many ECG studies [18, 6, 4, 10, 19], we included questions that inquire about various ECG symptoms in the ECG-QA dataset. To ensure the dataset quality, we excluded attributes with a low number of positive ECG samples in the test split, such as WPW (wolf-parkinson-white syndrome), resulting in a final selection of 64 attributes, including the 5 superclasses. Furthermore, we developed a regular expression parser to extract the grounded lead position of form-related symptoms from the ECG reports (See Supplementary B.1). This enables us to include questions in the ECG-QA dataset that
specifically address the leads in which symptoms are detected (e.g., Does this ECG show symptoms of inverted T-waves in lead I?), making the ECG-QA dataset more comprehensive.
Noise Considering that ECG measurements involve placing electrodes on specific body surfaces, it is inevitable to encounter various signal interferences such as baseline drift, which can be caused by patient movement or machine issues. Therefore, it becomes crucial to differentiate these interferences from the original ECG signals during analysis. To reflect this aspect in the ECG-QA dataset, we leveraged the signal noise information available in the PTB-XL metadata. This information is provided as a string indicating the specific lead positions where each noise is detected (e.g., "v1-v6" or "i-iii"). We parsed these strings to identify the exact lead positions associated with four different types of noises: Baseline drift, Static noise, Burst noise, and Electrode problems.
Stage of infarction Since identifying the stage of myocardial infarction (MI) helps healthcare professionals determine the most appropriate management strategies by assessing the risk profile for the patient, we also have considered the stage of infarction as an important attribute in the ECG-QA dataset. In the PTB-XL dataset, it distinguishes the stage of MI into six levels including intermediate stages: "I," "I-II," "II," "II-III," "III," and "Unknown." In addition, because there are two fields indicating the stage of infarction (infarction_stadium1 and infarction_stadium2), the statements could be potentially multiple. For the sake of simplicity, we simplified the stages into 4 levels by regarding the intermediate stages ("I-II" and "II-III") as their "lower" stage ("I" and "II"). Then, we used the second statement if there are multiple entries at a time. After defining an additional stage called "None" for those who do not have MI, we could derive five attributes for the stage of infarction.
Extra systole Since extra systoles can be a sign of underlying cardiac conditions or abnormalities, it is also important to detect them to evaluate the patient's heart health. To address the presence of extra systoles in the ECG-QA dataset, we utilized the relevant annotations provided in the PTB-XL metadata, which includes information about the occurrence of different types of extra systoles: Extrasystoles, Ventricular extrasystoles, and Supraventricular extrasystoles.
Heart axis The heart axis provides valuable information about the direction of the heart's electrical activity during each cardiac cycle. In the ECG-QA dataset, we have considered the heart axis as another crucial attribute since it is an important parameter that can help in diagnosing certain cardiac conditions. Although the PTB-XL metadata includes heart axis information, we did not utilize it because it does not specify the actual numerical values of the heart axis. Instead, we manually calculated the heart axis degrees by employing an external tool, NeuroKit2 [16]  $^{3}$ . Then, we classified them into four categories following the conventional standards: Normal heart axis, Left axis deviation, Right axis deviation, and Extreme axis deviation.
Numeric feature The ECG-QA dataset also incorporates numeric features that provide further insights into the cardiac signals. Similar to the heart axis, we used NeuroKit2 to calculate the numeric values for these features since the PTB-XL dataset does not explicitly provide such information. Specifically, we extracted the locations of P, Q, R, S, and T waves for each beat present in lead II, and computed six different numeric features: RR interval,  $P$  duration, PR interval, QRS duration, QT interval, and QT corrected. Given that a 10-second ECG recording typically contains multiple beats and thus multiple numeric values for each feature, we represented each feature using its median value. This approach helps to minimize the impact of abnormal contractions, such as ventricular premature contractions, on the calculated values. Additionally, we categorized each numeric value as below, within, or above the normal range, where the normal range criteria are described in Supplementary B.2, derived from a previous study [27]. These ranges serve as a useful reference for assessing the numerical measurements and identifying potential abnormalities in the cardiac signals.
3.1.2 Question type definition
We have defined two different types of questions. The first type pertains to the type of ECGs associated with a question, which can be categorized as follows: 1) Single, which refers to questions involving a single ECG; 2) Comparison-Consecutive, which involves comparison questions between two
This software is licensed under MIT license.
Table 2: Test performances for different question types. We also provide  $95\%$  confidence interval across 3 random seeds. The best performances for each question type are highlighted with boldface.
Question Type	per Q-type majority	M3AE† [2]	MedViLL† [17]	Fusion Transf.	Blind Transf.	Deaf Transf.
EM Acc.	AUROC	EM Acc.	AUROC	EM Acc.	AUROC	EM Acc.	AUROC	EM Acc.	AUROC
S-Verify	67.7	74.6±0.4	0.761±0.002	73.9±0.5	0.768±0.011	72.1±0.5	0.725±0.008	67.7±0.0	0.629±0.008	67.3±0.2	0.613±0.003
S-Choose	31.2	57.1±0.8	0.850±0.002	54.1±0.8	0.839±0.001	46.4±0.4	0.797±0.007	31.0±0.1	0.529±0.006	31.4±0.0	0.786±0.011
S-Query	23.2	41.0±0.5	0.836±0.002	40.4±0.6	0.831±0.004	37.4±0.6	0.791±0.011	24.0±0.0	0.549±0.006	27.0±0.1	0.754±0.006
CC-Verify	62.8	75.5±0.2	0.792±0.002	74.3±2.6	0.778±0.047	71.9±0.6	0.760±0.003	65.7±0.8	0.610±0.001	59.5±0.6	0.510±0.009
CC-Query	16.9	20.1±1.6	0.808±0.003	22.0±1.3	0.816±0.003	18.4±1.3	0.781±0.003	16.9±0.0	0.568±0.023	16.9±0.1	0.693±0.012
CI-Verify	66.1	75.3±0.9	0.769±0.010	77.5±1.6	0.823±0.021	68.1±0.6	0.723±0.010	66.2±0.1	0.508±0.004	61.1±0.5	0.505±0.004
CI-Query	1.10	4.19±0.2	0.741±0.008	3.50±0.2	0.758±0.004	2.19±0.1	0.704±0.004	0.95±0.1	0.527±0.008	1.11±0.0	0.632±0.016
S: Single, CC: Comparison-Consecutive, CI: Comparison-Irrelevant
consecutive ECGs from the same patient; and 3) Comparison-Irrelevant, which involves comparison questions between two irrelevant ECGs from different patients. Although Comparison-Irrelevant questions may not seem realistic in a clinical setting, we included these questions since they can help to reinforce a machine's comprehension ability and be utilized for model evaluation when comparing two different ECGs. In addition, inspired by GQA [8], the second type of question refers to the main function it should perform. These can be categorized as follows: 1) Verify, which corresponds to yes/no questions; 2) Choose, which applies to questions where the selection is made from two given options; and 3) Query, which are open-ended questions that seek to retrieve specific attributes.
By combining these two types, we could derive a total of 9 possible question types. However, we did not include the combinations of Comparison and Choose types since it seemed unnatural to select from two given options when comparing different ECGs. Similarly, with regards to the attribute types, we only considered SCP code and Numeric feature for comparison questions because these two attribute types are providing the most informative features when comparing two ECGs.
3.1.3 Paraphrase generation
To enhance the lexical diversity of the ECG-QA dataset, we manually curated paraphrases for each question template based on the machine-generated candidates by utilizing OpenAI's ChatGPT. We ensured that the questions in the test split were not included in the training set to evaluate the generalizability of the QA models on different lexical variations. The detailed procedure for generating paraphrases and its results are presented in Supplementary A.3.
3.2 QA sample collection
As shown in Figure 2 (b) and (c), we collected questions by plugging the corresponding attributes into the placeholder that existed in the question templates. For example, a question template "Does this ECG show symptoms of  $\{\mathsf{scp\_code}\} ?$  can be transformed into "Does this ECG show symptoms of conduction disturbance?" We further gathered the corresponding answers for each question and paired them to create (question, answer) pairs. Then, for each (question, answer) pair, we again randomly sampled the corresponding ECGs from the candidate ECGs. In each split, the candidate ECGs can be 1) all the single ECGs for Single questions; 2) all the  $(\mathrm{ECG}_1,\mathrm{ECG}_2)$  pairs where  $\mathrm{ECG}_1$  and  $\mathrm{ECG}_2$  are the consecutive ECGs from the same patient for Comparison-Consecutive questions; and 3) all the  $(\mathrm{ECG}_1,\mathrm{ECG}_2)$  combinations where  $\mathrm{ECG}_1$  and  $\mathrm{ECG}_2$  have different patient IDs for Comparison-irrelevant questions. After we finally replaced the template question with randomly selected paraphrases that matched the corresponding question's template, the process of collecting QA samples was complete. The detailed sampling strategies for different question types are described in Supplementary B.3.
After all these processes, the ECG-QA dataset consists of 267,539 training samples, 64,663 validation samples, and 82,146 test samples, which cover various types of attributes and questions. More detailed statistics of the dataset are described in Supplementary B.4.
Task formulation We formulate QA task as a multi-label classification over all possible answer options that exist in the ECG-QA dataset. The answer labels are composed of 88 attributes from the six attribute sets, 12 lead positions (i.e., lead I - lead V6), and 3 answers for Verify questions (yes, no, not sure), leading to a total of 103 answers. Note that we processed "None" answer as an empty label.
Baselines We implemented the following QA baselines:  $\mathbf{M}^3\mathrm{AE}^\dagger$  [2], MedViLL† [17], Fusion Transformer, Blind Transformer (seeing questions only), and Deaf Transformer (seeing ECGs only). Because the original implementations of  $\mathbf{M}^3\mathrm{AE}$  and MedViLL were intended to pre-train images with texts, we modified them to be applied to ECGs instead of images and pre-trained them using ECG data, as marked with †. Additionally, similar to the per Q-type prior in VQA [1], we include a prior model, per Q-type majority, which outputs only the most frequent answer for each question type in the test split. More details about each model implementation including training hyperparameters are described in Supplementary C.1.1.
4.1 Evaluation metrics
Exact match accuracy To calculate the exact match accuracy, we applied a threshold value of 0.5 to each score in the model's output vector,  $\hat{\mathbf{y}}\in \mathbb{R}^{103}$ , which gives a multi-hot vector of length 103. Then, we compare the output vector with the ground truth answer vector. If the two vectors are exactly the same, we assign a score of 1; otherwise, we assign a score of 0. To obtain the overall accuracy, we sum the scores for all the test questions and divide the aggregated score by the total number of questions, yielding the percentage of questions that were answered exactly.