ECG-QA: A Comprehensive Question Answering Dataset Combined With Electrocardiogram
Authors: Jungwoo Oh
Gyubok Lee
Seongsu Bae
Joon-myoung Kwon
Edward Choi
Venue: NeurIPS 2023 Datasets and Benchmarks Poster (2023)
Keywords: healthcare
question answering
electrocardiogram
ECG
ECG QA
TL;DR: The first question answering dataset specifically designed for electrocardiogram.

Abstract: Question answering (QA) in the field of healthcare has received much attention due to significant advancements in natural language processing. However, existing healthcare QA datasets primarily focus on medical images, clinical notes, or structured electronic health record tables. This leaves the vast potential of combining electrocardiogram (ECG) data with these systems largely untapped. To address this gap, we present ECG-QA, the first QA dataset specifically designed for ECG analysis. The dataset comprises a total of 70 question templates that cover a wide range of clinically relevant ECG topics, each validated by an ECG expert to ensure their clinical utility. As a result, our dataset includes diverse ECG interpretation questions, including those that require a comparative analysis of two different ECGs. In addition, we have conducted numerous experiments to provide valuable insights for future research directions. We believe that ECG-QA will serve as a valuable resource for the development of intelligent QA systems capable of assisting clinicians in ECG interpretations.

ECG-QA: A Comprehensive Question Answering Dataset Combined With Electrocardiogram
Jungwoo Oh', Gyubok Lee', Seongsu Bae', Joon-myoung Kwon', Edward Choi KAIST, Daejeon' Medical AI Inc., Seoul
fojw0123, Eyu,ok.lee, seongsu, edwardchoi) ac. kr. cto@medicalai. com2
Abstract
Question answering (QA) in the field of healthcare has received much attention due to significant advancements natural language processing. However, existing healthcare QA datasets primarily focus medical images, clinical notes, struc- tured electronic health record tables. This leaves the vast potential of combining electrocardiogram (ECG) data with these systems largely untapped. To address this gap, we present ECG-QA, the first QA dataset specifically designed for ECG analysis. The dataset comprises total of 70 question templates that cover wide range of clinically relevant ECG topics, each validated by an ECG expert to ensure their clinical utility. As result, dataset includes diverse ECG interpretation questions, including those that require a comparative analysis of two different ECGs. In addition, we have conducted numerous experiments provide valuable insights for future research directions. We believe that ECG-QA will serve as valuable resource for the development of intelligent QA systems capable assisting clinicians in ECG interpretations.
Introduction
In recent years, significant advancements natural language processing have revolutionized the field of question answering (QA) in wide range of domains. Previous works have demonstrated the great potential of QA systems in various domains, where they have been combined with different modalities such as images 32 15] & tables with images 24 [3.. Concurrently, QA systems have also been explored in the healthcare domain, including visual QA with chest X-ray [1 14],
QA 20), and QA over structured electronic health record (EHR) data 28 12
These pioneering efforts have successfully bridged the gap between general-domain QA and the medical field, unlocking possibilities to improve healthcare outcomes and enhance medical decision-making processes.
Despite this remarkable progress, there is noticeable absence of datasets that combine electro- cardiogram (ECG) data with question answering. As fundamental diagnostic tool in cardiology, ECG provides critical insights into the electrical activity of the heart and plays important role in detecting various cardiac conditions [3, [22 33). Consequently, integrating ECG data with QA systems holds tremendous potential improve the interpretation of cardiac data, leading more accurate diagnoses and personalized treatment plans.
To this end, we present ECG-QAT a novel QA dataset that incorporates ECG data for question answering tasks. To the best of our knowledge, ECG-QA is the first dataset that combines QA and ECG, opening up new avenues for integrating multi-modal machine learning with cardiac healthcare.
The main contributions of this work threefold:
'The datasct available at
licensed under CC-B B-400 license.
37th Conference on Neural Information Processing Systems (NeurIPS 2023) Track on Datasets and Benchmarks.

---

(a) Questions to a single ECG
Nave
Suye of InGrasion
Heart
Nanarii Seatars
40G degres
ahone banetin
Mhat interstie PCG
Doou Br CCD sice ausericlly
Whararndon this devited
Does de Intervel
sbove
Wether of there
None
Nirmalaas aa
(b) Comparison questions different ECGs
^
Code
Moneric feetce
(Wol
Compured
U3 conppared to prodiss oos?
ruo Ui RR interval the secunt bucting Decere normal
Uegrew
Figure 1: Sample question-answe answer pairs in ECG-QA. (a) Questions to a single ECG with various types of attributes. (b) Comparison questions between two different ECGs. Refer to Section B.1.1 for more details about each attribute type.
We propose the ECG-QA dataset, diverse collection of questions focused on ECG interpretation and analysis (See Fi This dataset introduces the novel concept of incorporating question answering into the realm of ECG analysis, making it valuable resource for developing and evaluating QA systems in the context of cardiology.
To cover more complex yet clinically critical questions, we include questions that require compar- ative analysis of two ECGs (See Figure (b)). This inclusion brings new degree of complexity, as addressing these comparison questions extends beyond conventional scope of ECG analysis using machine learning. By incorporating these types of questions, not only address the real-world needs of medical professionals but also broaden the potential applications machine learning in ECG analysis.
We provide benchmark for QA models, including recent large language models (LLMs), on the ECG-QA dataset, promoting further research and encouraging the development of novel methods to leverage ECG signals for question answering tasks. We believe that ECG-QA will serve as valuable resource in advaneing machine learning applications in cardiology and improving medical decision-making processes.
2 Related works
Medical QA datasets QA systems have been extensively explored in the healthcare domain, catering to the specific needs and challenges of medical data. However, most existing medical QA datasets are primarily based on clinical texts, medical images, or structured EHR tables. For example, Pampari al. [20] proposed the emrQA dataset, consisting of guestion-ansser pairs derived from unstructured clinical notes. In addition, Kovaleva et al. M and Liu et al. [4] proposed datasets for visual QA using X-ray images, aiming to investigate intelligent interactive systems for radiology. Meanwhile, in the field of QA over structured EHR data, Wang et al. [28) and Lee et al. [12 have developed datasets called MIMICSQL and EHRSQL respectively, which consist of questions and their corresponding SQL. queries. While these healthcare QA systems demonstrate the potential of leveraging medical data with QA to improve healthcare outcomes, there is currently no dedicated QA dataset specifically designed for ECG data despite its widespread use in diagnosing cardiovascular conditions and monitoring patients' heart health.
Electrocardiogram Previous studies in the field of ECG have predominantly focused on using ECG
data for diagnostic purposes such as identifying various heart diseases. For instance, Nejedly et al.
18 proposed an ensemble of residual networks with attention modules to classify cardiac diseases using ECGs, which won first place in the PhysioNeu/Coppuing in Cardiology Challenge 202/ (21). At the same challenge, Han et al. [6] achieved second place by utilizing SE-WRN [3], which is

---

Table 1: Sample template questions for different question & attribute types in ECG-QA.
Question type
Attribute type
Example template question
SCP Code
Does this ECG show symptoms of non-specifie ST changes?
Noise
Does ECG show baseline drift lead
Stage of infarction
this ECG show early stage of myocaridal infarction?
Single-Verit
Extra systole
Does this ECG show ventricular extrasystoles?
Heart axis
Does this ECG show left axis deviation?
Numeric feature
Does the RR interval of this ECG fall within the normal range?
Which sympiom does this ECG show, conduction disturbance or hypertrophy?
SCP Code
Noise
Which noise does this ECG show, baseline drift or static noise?
Which stage of infarction is this BCG at, early stage of myocurdial infarction or late stage of myocardial infarction?
Stage infarction
Single-Choose
Which kind of extra systoles does this ECG show, ventricular extrasystoles or supraventricular extrasystoles?
Extra systole
Which cardiac axis does this ECG show, left axis deviation right axis deviation?
Heart axis
Which range does the RR interval of this ECG fall in,
Numeric feature
below the normal range or within the normal range?
SCP Code
What form-related symptoms does this BCG show?
Noise
What kind of noises does this ECG show in lead I?
Stage of infarction
What stage of infarction is this ECG at?
Single-Query
Extra systole
What kind of extra systoles does this ECG show?
Heart axis
What direction is this ECG deviated to?
Numeric feature
What range does
interval of this ECG fall in?
Compared to the previous tracing. has left ventricular hypertrophy been
SCP
resolved in the recent tracing?
Comparison-
-Consecutive-eriily
Compared to the previous tracing, has the PR interval of the recent tracing become normal?
Numeric feature
What symptoms have been resolved in the recent tracing as compared to the previous one?
SCP Code
Compurison-
-Comsecutive-uurry
What numeric features of the recent tracing now have become normal compared to the previous one?
Numeric feature
Compared to the first ECG, has atrial fibrillation been newly detected the second ECG?
SCP Code
Comparison-
-Irrelevant- Verify
Compared to the first ECG, has the P duration of the second ECG changed to an abnormal value?
Numeric feature
SCP Code
What symptoms still remain the second ECG as compared to the first ECG?
Comparison- -Irelevant-Query
What numeric features of the sccond ECG are now considered abnormal values as compared to the first ECG?
Numeric feature
modules (7]. Furthermore, several works 4 J0 [9] studied self-supervised learning with ECGs to improve performances on the cardiac arrhythmia classification task. These works concentrate on classifying diagnoses based on single ECG, and do not consider the significance of comparing two ECGs, despite its importance in clinical contexts. For example, by detecting resolved symptoms after some treatments, the medical practitioners can assess the effectiveness of the treatments and evaluate the progress of the patient's s condition. To reflect this clinical reality, we have included questions that involve the comparison of two ECGs within the ECG-QA dataset, making our dataset unique and valuable.
combination of wide residual network (30 and squeeze-and-excitane
3 Dataset Construction
We constructed the ECG-QA dataset upon the PTB-XL dataset (26 which offers comprehensive metadata regarding ECGs annotated by expert cardiologists. This metadata covers a wide range of information including ECG reports, diagnostic statements, diagnosis likelihoods, and signal-specific properties. To ensure the high quality of our dataset, we performed additional filtering on the original
"Tii dataset is licensed under CC-BY-4.0 license.
3

---

(a) Attribute Type & Question Type
(d) Sampling
Attribute Type
Question Type
Zee Question Type
Sampled EGGs
SCP Code
Single
Verify
Yes: [10836, 10240, -:) No: (14899, 7665. -..
Notse
Companson-
Choase
CD: [23343, 13523, - HYP:[14798, 6206,
Heart axts
Consetutive
Query
Both: [5242, 954, Netther: [2931, 4380,
Stage Inforction
Compansor-
None: [LAFS, [308, 1977,-.] (L9H, ISC_
Extra systole
Numeric feoture
(b) Templatization
Quesstion Template
[c) Question & Answer Collection
Does this ECG show symptoms of seep_rade??
Template on
Anzwer
Which symptom does this ECG show, S/sop_ codel] or rree_]?
disturbance?
Does this LCG show symptoms conduction
Yes//o No
What diegnostic e symptoms does this ECG show, including uncertein symptoms??
conduction disturbance or hypertrophy?
which symptom does this ECG show.
CD / HVP Both Neither
the tracing, has _ocp_ been resalved tracing?
What diagnostic symptoms show, including symptamm?
None/
SC //-
Figure 2: Visualization of the ECG-QA sample generation pipeline. The numbers in the sampling stage (d) stand for the ECG IDs in the PTB-XI. dataset. In the sampling process, we also convert the template questions into pre-defined paraphrases for each sample.
PTB-XL dataset. Specifically, we selected ECGs that marked with val walidated__ by h_mmann tag set to True, which indicates the validation by a human cardiologist, and excluded ECGs that had empty reports. As a result, the ECG-QA dataset was constructed using 16, 054 samples of 10-second ECGs from the PTB-XL dataset. In addition, we split the samples into training and test sets according 8:2 ratio based on their patient IDs before generating QA samples prevent the overlapping of ECGs between training and test sets. We again split the training samples into training and validation sets by a 9:1 ratio, yielding 7.2:0.8:2.0 wiapiingglliiatio-ttee distribution.
3.1 Question template
To generate QA samples, we start by creating the question templates to collect questions, answers, and their corresponding ECGs. Because the questions are fully derived from these templates, it is important to define templates that are not only diverse but also clinically meaningful. To achieve this goal, we extracted the relevant attributes from the PTB-XI. metadata determine the content of the questions (i.e., attribute types) and categorized the questions into several question types. Then, combined these types construct the template questions, and additionally generated template paraphrases to add lexical diversity to our dataset. As a result, we defined total of 70 templates as shown in Supplementary A..]] and we also provide the example questions derived from these templates in Table [ All the processes of designing question templates have been validated by board-certified medical expert from the Department of Critical Care and Emergency Medicine in terms of clinical utility. The detailed processes of each step are described in the following subsections, as well visualized in Figure 2 (a) and (b).
3.1.1 Attribute type extraction
SCP code The PTB-XL. dataset provides SCP codes for each ECG sample, consisting of 71 different ECG symptoms that adhere to the SCP-ECG v0.4 that preceded the current SCP-ECG standard (23). These attributes are composed of form-related (e.g., inverted T-waves), rhythm-related (e.g., sinus arrhythmia), and diagnostic symptoms (e.g., non-specific ischemic) along with additional 5 superclasses for diagnostic labels. Given that detecting cardiac symptoms is a primary objective in many ECG studies
4 TO 19, we included questions that inquire about various ECG symptoms in the ECG-QA dataset. To ensure the dataset quality, we excluded attributes with a low number of positive ECG samples in the test split, such as WPW (wolf-parkinson-whiee syndrome), resulting in a final selection of 64 attributes, including the 5 superclasses. Furthermore, we developed regular expression parser to extract the grounded lead position of form-related symptoms from the ECG
[88 6
reports (See Supplementary B.I This enables to include questions in the ECG-QA dataset that

---

specifically address the leads in which symptoms are detected (e.g., Does this ECG show symptoms of inverted T-waves in lead I?), making the ECG-QA dataset more comprehensive.
Noise Considering that ECG measurements involve placing electrodes specific body surfaces, it is inevitable to encounter various signal interferences such as baseline drift, which can be caused by patient movement or machine issues. Therefore. it becomes crucial to differentiate these interferences from the original ECG signals during analysis. To reflect this aspect in the ECG-QA dataset, we leveraged the signal noise information available in the PTB-XL metadata. This information is provided as a string indicating the specific lead positions where each noise is detected (e.g. "v1-v6" or "i-iii"). We parsed these strings to identify the exact lead positions associated with four different types of noises: Baseline drift, Static noise, Burst noise, and Electrode problems.
Stage of infarction Since identifying the stage of myocardial infarction (MI) helps healthcare professionals determine the most appropriate management strategies by assessing the risk profile for the patient, we also have considered the stage of infarction as an important attribute in the ECG-QA dataset. In the PTB-XL dataset, it distinguishes the stage of MI into six levels including intermediate stages: "I," "I-II." "II," " "II-III" " "III," and "Unknown." " In addition, because there are two fields indicating the stage of infarction (infarction_stae stadiumi and infarction. sereeiionsttdiumnn stadiumm), the statements could be potentially multiple For the sake of simplicity, we simplified the stages into levels by regarding the intermediate stages ("I-II" and "II-III") as their "lower" stage ("I" and "II"). Then, we used the second statement if there are multiple entries at a time, After defining an additional stage called "None" for those who do not have MI, we could derive five attributes for the stage of infarction.
Extra systole Since extra systoles be sign of underlying cardiac conditions abnormalities, it is also important to detect them to evaluate the patient's 's heart health. To address the presence of extra systoles in the ECG-QA dataset, we utilized the relevant annotations provided in the PTB-XL metadata, which includes information about the occurrence of different types of extra systoles: Extrasyttoles, Ventricular extrasystoles, and Supraventricular extrasystoles.
Heart axis The heart axis provides valuable information about the direction of the heart's electrical activity during each cardiac cycle. the ECG-QA dataset, we have considered the heart axis as another crucial attribute since it is an important parameter that can help in diagnosing certain cardiac conditions. Although the PTB-XC XL metadata includes heart axis information, we did not utilize because it does not specify the actual numerical values of the heart axis. Instead, we manually calculated the heart axis degrees by employing an external tool, NeuroKit2 [T6] 3 Then, we classified them into four categories following the conventional standards: Normal heart axis, Left axis deviation, Right axis deviation, and Extreme axis deviation.
Numeric feature The ECG-QA dataset also incorporates numeric features that provide further insights into the cardiac signals. Similar to the heart axis, we used NeuroKit2 to calculate the numeric values for these features since the PTB-XL dataset does not explicitly provide such information. Specifically, we extracted the locations of P. Q R, S, and T waves for each beat present in lead II, and computed six different numeric features: RR interval, P duration, PR interval, QRS duration, QT interval, and QT corrected. Given that a 10-second ECG recording typically contains multiple beats and thus multiple numeric values for each feature, we represented each feature using its median value. This approach helps to minimize the impact of abnormal contractions, such as ventricular premature contractions, on the calculated values. Additionally, we categorized each numeric value as below, within, or above the normal range, where the normal range criteria are described in Supplementary [.2] derived from a previous study (27). These ranges serve as a useful reference for assessing the numerical measurements and identifying potential abnormalities in the cardiac signals.
3.1.2 Question type definition
We have defined two different types of questions. The first type pertains to the type of ECGs associated with question, which can be categorized as follows: 1) Single, which refers to questions involving a single ECG; 2) Comparison- Comparison-Coonseuutiee which involves comparison questions between two
This software is licensed under MIT license.
5

---

Table 2: Test performances for different question types. We also provide 95% confidence interval across random seeds. The best performances for each question type are highlighted with boldface.
Question
Quyyne
M' B
MedViLL [7]]
Fusion Transt.
Blind Transf.
Deaf Tiansf.
Type
majunity
EM Acc.
AUROC
EM Aec.
AUROC
EM Acs.
AUROC
EM Acc.
AUROC
EM Acc.
AUROC
S-Vendy
67.7
74.6.0
0,761
70.9.a.
0.76
72.1
0. 0.725
67.74m6
67.
0.813_
S-Choose
24.1.6.
2.838
0.707.200
$1.0an3
31.4
0.785-
S-Query
41.0.os
0,858...02
40.4
a.534
97.4
0,701
24.0anc
1.500
27.0ua.
0.754
CC-Venty
73.0.n3
0.7922n00a
74.3224
0.778a5e0
71.92n0
0.7000.0000
63-7an4
2.020
39.5-
0.520_
Cn-Quay
16.0
20.1214
2080
22.0,1.3
08066.0.00
18.4215
2.7812000
16.9nm
16.9a..1
0.003
Venlyy
66.1
75.8.0
0.760
77.5a1.4
88.1anc
0. 0.728
06.2
0.508 10.00
0.508
6l.1
0.500
Ci-Query
1.10
4.19,09
0.741go0ms
3.50g5.
0. 0.708 nos
2.19g0
0.704goo0
0.30gm
0.027
1.11g90
0.588226.30
Single, Compatsonn Comsecutve, CI: Comparsonn Irrelvant
consecutive ECGs from the same patient; and 3) Comparison-Irrlleaan,, which involves comparison questions between two irrelevant ECGs from different patients. Although Comparison-Irreler questions may not seem realistic in clinical setting, we included these questions since they can help reinforce a machine's 's comprehension ability and be utilized for model evaluation when comparing two different ECGs. In addition, inspired by GQA [8) the second type of question refers to the main function it should perform. These can be categorized as follows: 1) Verify, which corresponds to yes/no questions; 2) Choose, which applies to questions where the selection is made from two given options: and 3) Query, which are open- questions that seek to retrieve specific attributes.
By combining these two types, we could derive a total of 9 possible question types. However, we did not include the combinations of Comparison and Choose types since it seemed unnatural to select from two given options when comparing different ECGs. Similarly, with regards to the attribute types, we only considered SCP code and Numerie feature for comparison questions because these two attribute types are providing the most informative features when comparing two ECGs.
3.1.3 Paraphrase generation
To enhance the lexical diversity of the ECG-QA dataset, manually curated paraphrases for each question template based on the machine- machine-generated candidates by utilizing OpenAl's ChatGPT. We ensured that the questions in the test split were not included in the training set to evaluate the generalizability of the QA models on different lexical variations, The detailed procedure for generating paraphrases and its results are presented in Supplementary A.3
3.2 QA sample collection
As shown in Figure 2 (b) and (c), we collected questions by plugging the corresponding attributes into the placeholder that existed in the question templates. For example, question template "Does this ECG show symptoms of $ [(sp__code) of conduction disturbance? ?" We further gathered the corresponding answers for each question and paired them to create (question, answer) pairs. Then, for each (question, answer) pair, we again randomly sampled the corresponding ECGs from the candidate ECGs. In each split, the candidate ECGs can be 1) all the single ECGs for Single questions; 2) all the (ECG,. ECG2) ) pairs where ECG, and ECGg are the consecutive ECGs from the same patient for Comparison Consecutive questions; and 3) all the (ECG,, FCG2) combinations where ECG, and ECG₂ have different patient IDs for Comparison--rreleevan questions. After we finally replaced the template question with randomly selected paraphrases that matched the corresponding question's 's template, the process of collecting QA samples was complete. The detailed sampling strategies for different question types are described in Supplementary B.3
can be transformed into "Does this ECG show symptoms
After all these processes, the ECG-QA dataset consists of 267.539 training samples, 64,663 validation samples, and 82, 146 test samples, which cover various types of attributes and questions. More detailed statisties of the dataset are described in Supplementary B.4
4 Experiments
Task formulation We formulate QA task as a multi-label classification over all possible answer options that exist in the ECG-QA dataset. The answer labels are composed of 88 attributes from the six attribute sets, 12 lead positions (i.e., lead I lead V6), and 3 answers for Verify questions (yes, no, not sure), leading total of 103 answers. Note that we processed "None" answer an empty label.

---

Baselines We implemented the following QA baselines: AE 2. MedViLL 17 Fusion Transformer, Blind Transformer (seeing questions only), and Deaf Transformer (seeing ECGs only). Because the original implementations of A and MedViLL. were intended to pre-train images with texts, we modified them to be applied to ECGs instead of images and pre-trained them using ECG data, as marked with T. Additionally, similar to the per Q-type prior in VQA [1, we include a prior model, per Q-type majority, which outputs only the most frequent answer for each question type in the test split. More details about each model implementation including training hyperparameters are described Supplementary C.I.I
4.1 Evaluation metries
Exact match accuracy To calculate the exact match accuracy, we applied a threshold value of 0.5 each score the model's output vector, y E IR 1200 which gives a multi-hov hot vector of length 103. Then, we compare the output vector with the ground truth answer vector. If the two vectors are exactly the same, we assign a score of I; otherwise, we assign a score of To obtain the overall accuracy, we sum the scores for all the test questions and divide the aggregated score by the total number of questions, yielding the percentage of questions that were answered exactly.
AUROC While the exact match accuracy is a useful metric, it may not fully capture the model's performance since it does not consider partial credits, especially in the case of Query questions that require consideration of much more attributes. To provide a more comprehensive evaluation, we employ the area under ROC curve (AUROC) as another metric. When calculating AUROC, adopt a cautious approach by only considering the "valid" answer candidates for each question. This approach aims prevent overestimation, as the model might naturally assign lower scores "invalid" answer options. For example, in question like "Which noise does this ECG show, baseline drift or static noise?", we exclusively consider the scores of the answer options baseline drift and static noise. We collect scores for each answer option over all the samples and compute macro-averaged AUROC among the answer options.
4.2 Upper bound experiments
In the field of clinical medicine, even experienced medical practitioners cannot be entirely certain when making crucial decisions, such as diagnosing patient's condition. Similarly, the ECG-QA dataset can also suffer from this inherent uncertainty even though we extracted attributes from the existing annotations made by expert cardiologists. As 100% accuracy is unlikely to attain due the inherent uncertainty, we aim to estimate the upper bound performance a model can achieve with our dataset, and use
reference when evaluating the model performance.
Within our dataset, we speculate that questions of the Single-Verify type necessitate basic perceptual abilities while other question types can be solved by logically combining these perceptual abilitics. For example, Single-Choose questions can be answered by verifying the presence of each attribute in the given two options, and similarly, Single-Query questions can be solved by verifying the presence of each element within the specified attribute set. Consequently, we hypothesize that achieving high performance
the whole ECG-QA dataset is unlikely without a high level of perceptual ability. Based on this hypothesis, we can estimate the upper bound performances for the whole ECG-QA dataset by measuring the upper bound performances of the Single- samples. To this end, designed the following experiments.
We convert all the Single QA samples (Single-Veriee Single-Choose, Single-Query) in the training set into the format that ECG classification models can process, and train the classification models using the converted training samples. Similarly, after converting the Single- Veriff samples in the test set, we estimate the upper bound performance by measuring performances on the converted Single-Verite test samples. Then, compare this upper bound with Single- Verify performances of QA models to show how much the QA baselines can be improved in terms of their perceptual ability. The detailed process of converting QA samples into ECG classification format is described in Supplementary C.1.2
For these experiments, we employ powerful ECG classification models classifying all the individual attributes present in the Single-Verse samples. The models we use include a Transformer-bases model pre-trained with the W2V+CMSC+RLM [19] method, Resnet with Attention [18), and SE-
7

---

Table Macro- -verr-ae test performances of up- per bound models over all attributes for Single- Verify questions
Table 4: Macro-averaged test performances of QA models over all attributes for Single- Verify questions.
Upper bound Model
Acc.
AUROC
QA Model
Acc.
AUROC
W2V+CMSC+RLM [19
83.0 0.86440.00s
M AEt [2
80.8
0.808 00000
Resnet-Attention [8]
82.6=0.3
0.875+0.002 002
MedViLL [17
79.8+0.3
0.809
SE-WRN 6
83.1
0.88340.002
MAX
85.4
0.907 20.002
Fusion Transf.
76.4
0.764=0.010
WRN [6]. In addition, to present the maximized upper bound, we derive another model that takes only the maximum score among the three models for each attribute, which is denoted as MAX. Detailed model implementations and training configurations are presented in Supplementary C.1.2
4.3 Modeling with LLMs
As for one of the future research directions with our dataset, we further investigated the possibility of leveraging LLMs for ECG-QA. Inspired by ChatCAD [29], for each QA sample, we transformed ECGs into text descriptions using the output from the trained upper bound model (SE-WRN) and forwarded them to several OpenAl's GPT models (gpt-4, gpt-3. 3 5-turbo, text text-davinci-oos along with the corresponding question. Due the restricted quota of OpenAl's api usage policy, we randomly sampled 10% from the ECG-QA test set and conducted experiments only once for each LLM model. The detailed processes including prompts that we used are described in Supplementary C.1.3
4.4 Results
QA results The baseline results are presented in Table 2 We also report test performances for different attribute types Supplementary C.2 As expected, Blind and Deaf Transformer exhibit poor performance while other models all achieve higher scores compared to the prior model (per Q-type majority), indicating that our dataset cannot be solved by solely seeing each question and ECG separately. Furthermore, among the top three models AEE MedViLL and Fusion Transformer), the pre-trained models (M AE and MedViLL') outperform Fusion Transformer, which demonstrates the potential advantages of utilizing novel multi-modal pre-training methods for our dataset. Additionally, the lower performance of Choose or Query questions compared to Verify questions suggests that the primary challenges in our dataset lie model's ability to learn logical and set operations based on the basic perceptual abilities that can be acquired from Verify questions.
Upper bound results The results of the upper bound experiments are reported in Table 3 and 4 When we compare SE-WRN, which is the best upper bound model, with the best QA model, M M AE we can see that the perceptual ability of baseline models can be improved by 2. 2.%%p and 7.5%p in terms of EM accuracy and AUROC, respectively. Moreover, when comparing with MAX, which is expected to show a higher upper bound, the differences are increased 4.6%p in EM accuracy and 9.9%p in AUROC. We believe these upper bound results can serve as a useful yardstick for assessing the basic perceptual ability required for more complicated questions such as Choose or Query.
LLM modeling results The results of the experiments with LLMs are presented in Table 5 Interestingly, the performance of all the GPT models did not surpass that the QA baseline model. We speculate that this is due to two primary reasons: 1) the upper bound model (i.e. SE-WRN fails to accurately extract necessary information, and 2) some questions were too complicated to be answered with zero-shot prompt. Since LLMs fully rely the ECG classification model for interpreting the ECGs, their performance inevitably depends on the capabilities of the ECG classification model. However, we cannot guarantee that SE-WRN is such a perfect model that always outputs accurate interpretations, because it has been trained with a limited set of QA training set to measure only the upper bound of the perceptual ability. Therefore, we expect significant performance gains if we have
As all of these experiments were conducted prior to June 7th, 2023, the associated models were referencing the earlier legacy versions. More precisely, at the time of the experiments, gpt-4 corresponded to gpt-4-0314. and gpt-3, 5-turoo matched with gpt-3, 5.55-tuu ttrbbo-00010

---

Table 5: Test EM accuracies for different question types. Note that randomly sampled 10% from the ECG-QA test set for each question type test the models due to the restricted quota of OpenAl's api usage policy. The best performance for each question type are highlighted with boldface.
Question Type
per Q-type majority
SE-WRN
SE-WRN
SE-WRN
M 22
+ gpt-4
+ gpt-3.5 3-tubbo text-davinci--003
Verify
69.1
71.0
69.3
75.0
76.0
S S-Choose
30.7
48.1
36.1
37.8
58 58.2+0..
S Query
25.3
35.7
31.1
36.0
40.0+1.6
CC- CerVyyyyy
58.4
54.9
58.2
56.3
74.7 11.2
CC-Query
17.3
13.0
10.5
15.4
21.2+2.0
CI- Verify
67.0
68.8
64.1
71.5
75.211.7
CI-Query
1.32
2.53
1.32
1.40
4.3640.7
S: Single, CC: Comparrison CI: Compuniion--mmllevat
a strong classification model that can extract all the existing information from an ECG, and fine-tune LLMs with our QA training set (iee,, applying instruction learning LLMs).
Conclusion
In this work, present ECG-QA, the first QA dataset that incorporates ECG data for question answering tasks. Our dataset is designed to ensure clinical relevance and has been validated by an ECG expert. We created carefully designed question templates, which leverage clinically meaningful attributes extracted from the PTB-XL dataset, to generate a diverse collection of questions, including those that require the comparison of two ECGs. We believe that our dataset has the potential to significantly advance the field of ECG question answering research and contribute to the improvement of clinical practice analyzing ECG data.
As for the future research directions with ECG-QA, one the promising avenues is the exploration of multi-modal LLMs that can simultancously process both ECG signals and natural language. While there is extensive work on LLMs that combine vision and language, there has been limited research on models that integrate signal processing with natural language. We believe our dataset can serve as an excellent testbed for such models.
6 Limitation
Despite our best efforts to create the current version of the dataset, there are some limitations as follows.
Small number of ECGs Due to the limited number of ECGs available in the original dataset (PTB-XL), our dataset was constructed using relatively small number of ECGs (~16k), which leads that questions involving too rare symptoms (e.g.. Wolf-Parkinson White syndrome) could not be included. To provide more diverse combinations of ECGs and questions by incorporating questions regarding very rare attributes, we are planning to employ another dataset that is larger than the PTB-XL dataset such as MIMIC-IV-ECG [. which is planned to be released late 2023.
Upper-bound of the dataset As mentioned in Section 4.2 given the intricacies of the medical field, even medical experts cannot provide 100% accurate diagnoses for all questions. Thus, the upper- bound of the dataset itself is not expected to be 100%. To address this, conducted experiments demonstrating the estimated upper-bound for each question type and attribute.
Old version SCP-ECG standard Despite SCP-ECG v3.0 being the latest version, the metadata of the original dataset, PTB-XL, follows the SCP-ECG v0.4 standard. Consequently, in ECG-QA, we were constrained to categorize various symptoms based on the SCP-ECG v0.4 standard. However, after investigating how the SCP codes in SCP-ECG v3.0 are categorized, we found that there is only little difference between SCP-ECG v0.4 and v3.0 regarding the SCP codes used in PTB-XI., Among the SCP codes used in PTB-XC X-XL, only one SCP code (BIGU, bigeminal pattern unknown origin,

---

SV or Ventricular) has a different representation, which has changed to "SVBIG" (supraventricular bigeminy BIGU bigeminal patter unkown origin, SV Ventricular) in SCP-ECG v3.0. The rest of the SCP codes have maintained their codes and definitions intact in SCP-ECG v3.0. Therefore, we helieve that the impact of the differences between the two versions will not be significant in the ECG-QA dataset.
Automatic generation paraphrases Although the paraphrases were manually curated, initial candidates were automatically generated by ChatGPT, which may not be optimal strategy. We expect that paraphrases could have been more diverse if we had involved medical practitioners in manually generating paraphrases.
Acknowledgments and Disclosure of Funding
This work was supported by Institute of Information & communications Technology Planning & Evaluation (ITTP) grant (N0.2019-0-00075), National Research Foundation of Korea (NRF) grant 13A2220 100945), and Korea Medical Device Development Fund grant (Project Number: 171113816 160, KMDF PR 20200901 _009),, funded by the Korea government (MSIT, MOTIE, MOHW, MFDS).
10

---

References
[1] Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C. Lawrence Zitnick, and Devi Parikh. Vqa: Visual question answering. In Proceedings of the IEEE international conference on computer vision, pages 2425-2433, 2015.
[2] Zhihong Chen, Yuhao Du Jinpeng Hu, Yang Liu, Guanbin Li, Xiang Wan, and Tsung- Hui Chang. Multi- modal masked autoencoders for medical vision-and- -ooo--an--angguge pre- pre-training. Medical Image Computing and Computer Assisted Intervention-MiC MMCCII 2022: 25th International Conference, Singapore, September 18-2 22, 2022, Proceedings, Part V. pages 679-689. Springer, 2022.
[3] W Bruce Fye. A history of the origin, evolution, and impact of electrocardiography The American americanjournal of cardiology, 73:139937-994 1994.
[4] Bryan Gopal, Ryan Han, Gautham Raghupathi, Andrew Ng. Geoff Tison, and Pranav Rajpurkar. 3kg: contrastive learning of 12-lead cloetrocardiograms using pyyssilogicalll-inspiii augmentations. In Machine Learning for Health, pages 55-1677 PMLR, 2021.
[5] Brian Gow, Tom Pollard, Larry A Nathanson, Alistair Johnson, Benjamin Moody, Chrystinne Fernandes, Nathanicl Greenbaum, Seth Berkowitz, Dana Moukheiber, Parastou Eslami, ct al. Mimie-i--cog-dagnostt electrocardiogram matched subset.
[6] Hyeongrok Han, Seongjae Park, Seonwoo Min, Hyyn-SSo Choi, Eunji Kim, Hyunki Kim, Sangha Park, Jinkook Kim, Junsang Park, Junho An, el. Towards high generalization performance on electrocardiogram classification. In 202/ Computing in Cardiology (CinC), volume 48, pages 1 1-4. IEEE, 2021.
[7] Jie Hu, Li Shen, and Gang Sun. Squeezz-and-eecitation networks. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 7132-7141, 2018.
[8] Drew A Hudson and Christopher D Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 6700-67777 6709, 2019.
[9] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint urXiv: 1412.6900,6 2014.
[10] Dani Kiyasseh, Tingting Zhu, and David A Clifton. Clocs: Contrastive learning of cardiac signals across space, time, and patients. In International Conference on Machine Learning, pages 5606-566 5615. PMLR, 2021.
[11] Olga Kovaleva, Chaitanya Shivade, Satyananda Kashyap, Karina Kanjaria, Joy Wu, Deddeh Ballah, Adam Coy, Alexandros Karargyris, Yufan Guo, David Beymer Beymer, Anna Rumshisky, and Vandana Mukherjee Mukherjee. Towards visual dialog for radiology. In Proceedings of the 19th SIGBioMed Workshop on Biomedical Language Processing, pages 60 69, Online. July 2020. Association for Computational Linguistics. doi: 10. 118836//12002// (1/2000.bionlpp. 1.6. URL
wrg.:lllooyor//2222200 bionlp-1
[12] Gyubok Lee, Hyeonji Hwang, Scongsu Bac, Yconsu Kwon, Woncheol Shin, Scongjun Yang, Minjoon Sco, Jong-Yes Joug-Yeup Kim, and Edward Choi. Ehrsql: A practical text-to-sql benchmark for electronic health records. Advances in Neural Information Processing Systems, 35:15589-15601, 2022.
[13] Yongqi Li, Wenjie Li, and Liqiang Nie. MMCoQA: Conversational question answering over text, tables, and images. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 4220-4231, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 10. 10.18653/v1/20222..e acl 290g URL htap:::///clannhooogg.ooom [290
[14] Bo Liu, Li-Ming Zhan, Li Xu, Lin Ma. Yan Yang, and Xiao-Ming Wu. Slake: A semantically-labeled knowledge-enhantt dataset for medical visual question answering. 2021 IEEE 18th International Symposium on Biomedicul Imaging (ISBI), pages 1650-1654. IEEE, 2021.
[15] Pan Lu, Lei Ji, Wei Zhang, Nan Duan, Ming Zhou, and Jianyong Wang. R-vqa: learning visual relation facts with semantic attention for visual question answering. In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 1880-1889, 2018.
[16] Dominique Makowski, Tam Pham, Zen J. Lau, Jan C. Brammer, François Lespinasse, Hung Pham, Christopher Schölzel, and S. H. Annabel Chen. NeuroKit2: A python toolbox for neurophysiological signal processing. Behavior Research Methods, 53(4); 1696, feb 2021. doi: 0ww0://--112222202--1-0000333110 URL https: : //doi. org/ 10. 375872F 020 01516-y

---

[17] Jong Hak Moon, Hyungyung Lee, Woncheol Shin, Young- Hak Kim, and Edward Choi. Multi-modal understanding and generation for medical images and text via vision-language pre-training. IEEE Journal of Biomedical and Health Informatics, 26(12):6070-608) 2022.
[18] Petr Nejedly, Adam Ivora, Radovan Smisek, Ivo Viscor, Zuzana Koscova, Pavel Jurak, and Filip Plesinger. Classification of ecg using ensemble of residual cnns with attention mechanism. 2021 Computing in Cardiology (CinC), volume 48, pages 1-4. IEEE, 2021.
[19] Jungwoo Oh, Hyunseung Chung, Joon- mooung Kwon, Dong-gyv gyun Hong, and Edward Choi. -aad-agnottic self-supervised learning for local and global representations of electnocardiogram. In Conference on Health, Inference, and Learning, pages 338-353, 353. PMLR, 2022.
[20] Anusri Pampari. Preethi Raghavan, Jennifer Liang, and Jian Peng. emrqa: A large corpus for question answering on electronic medical records. arXiv preprint 1x0iv:809,000 00732, 2018.
[21] Matthew Reyna, Nadi Sadr, Anaie Gu, Erick Andres Perez Alday, Chengyu Liu, Salman Seyedi, Amit Shah, and Gari Clifford. Will two do? varying dimensions in electrocardiography: the PhysioNed/Computin in Cardiology Challenge 2021. Computing in Cardiology 2021, 48:1-4, 2021.
[22] Anna Rosick and Krzysztof Leksowski. The risk factors and prevention of cardiovascular discase: the importance of electrocardiogram in the diagnosis and treatment of acute coronary syndrome. Therapeutics and clinical risk management, pages 1223 1029, 2016.
[23] Paul Rubel, Danilo Pani, Alois Schloegl, Jocelyne Fayn, Fabio Badilini, Peter W Macfarlane, and Alpo Varri. Sep-ceg v3. O: An enhanced standard communication protocol for computer-assisted clcctrocardiography, In 2016 Computing in Candiology Conference (CinC), pages 309-312, 312. IEEE, 2016.
[24] Alon Talmor, Ori Yoran, Amnon Catav, Dan Lahav, Yizhong Wang, Akari Asai, Gabriel Ilharco, Hannaneh Hajishirzi, and Jonathan Berant. Multimodalqa: Complex question answering over text, tables and images. arXiv preprint 2rXiv2214400003 2021.
[25] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pages 5998-6008, 2017.
[26] Patrick Wagner, Nils Strodthoff, Ralf-Dicter Bousseljot, Wojciech Samek, and Tobias Schaeffter. PTB- XL, large publicly available clectrocardiograpyy datasct (version 1.0.3). PhysioNet, 2022, doi: https: //doi.org/10. 1..00....10000 x3026/x4tdx9922.
[27] Li-ping Wang. Mi Shen, Jia-fei Tong, and Jun Dong. An uncertainty reasoning method for abnormal ecg detection. In 2009 IEFF International Symposium IT' in Medicine & Education, volume 1, pages 1091-1096. IEEE, 2009.
[28] Ping Wang, Tian Shi, and Chandan K Reddy. Text-to-sql generation for question answering on electronic medical records. In Proceedings of The Weh Conference 2020, pages 350-361, 2020.
[29] Sheng Wang, Zihao Zhao, Xi Ouyang, Qian Wang, and Dinggang Shen. Chatcad: Interactive computer- aided diagnosis on medical image using large language models. arXiv preprint arXiv:2302.072577 2023.
[30] Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. arXiv preprint arXiv: 1605.07146, 2016.
[31] Xian Zhong, Oubo Gong, Wenxin Huang, Lin Li, and Hongxia Xia. Squeeze-and endeecctttinnnnii wide residual networks image classification. In 2019 IEEE International Conference on Image Processing (ICIP), pages 395-399. IEEE, 2019.
[32] Yuke Zhu, Oliver Groth, Michael Bernstein, and Li Fei-Fei. Visual7w: Grounded question answering in images. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4995-5004, 2016.
[33] Peter J Zimethaum and Mark E Josephson. Use of the electrocardiogram in acute myocardial infarction. New England Journal of Medicine, 348(14) (48110)93339440 2003.
12


Reviews:
[{"review_id": "LXvh5rDYMu", "reviewer": "Reviewer_28Uj", "summary": "", "questions": "", "limitations": "The limitations have been stated clearly by the authors:\n- Due to the limited number of ECGs in the original PTB-XL dataset, our dataset was built using a relatively small number of ECGs (approximately 16k).\n- While the paraphrases were manually curated, the initial candidates were automatically generated by ChatGPT, which might not be the best approach. Paraphrases could have been more diverse if medical practitioners were involved in manually generating them.", "rating": 7, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "- The paper is well structured and the ideas are well presented.\n- The Dataset Construction section is detailed and authors provide the question generation pipeline clearly. \n- The paper provides replicable baselines. The experiments are supported by detailed Appendix (hyper-parameters, LLM prompting,...)\n- Authors propose an interesting Upper bound experiments ", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "clBKHFzFNp", "reviewer": "Reviewer_4ESf", "summary": "", "questions": "", "limitations": "The authors did not include a limitations section for their dataset. The authors should detail the limitations of their extraction of query questions and answering from the original dataset (if any), the limitations regarding human labels (which the article touched upon in Section 4.2), the limitations regarding the SCP-ECG v0.4 standard, as it has now been superseded, and any other limitations the authors have realized in the construction of the dataset and benchmarking. ", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "1. The authors provide one of the first QA datasets for ECG analysis, and the dataset is of a reasonable size and richness.\n2. Having QA questions where 2 ECGs are compared is crucially important and in the reviewer's opinion a major highlight of the provided dataset.\n3. The dataset repository is well-documented and there are clear instructions on how to reproduce and run experiments using the generated dataset. \n4. The experiments are well-designed and executed, and the writing is generally clear. ", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "wdi4Ot4HvF", "reviewer": "Reviewer_qtCQ", "summary": "", "questions": "", "limitations": "* It seemed that many of the questions were designed to be answered with a simple \"Yes\" or \"No\" or with a class classification. If this is the case, what is the qualitative difference between these easy-to-answer questions and simply solving a classification problem?", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "* A QA dataset with a variety of questions focused on ECG is rare and has potential clinical significance.\n* As a data set based on expert review, it is expected to be of high quality.\n* The chapter on \"question templates\" is particularly well written and would be useful for researchers wishing to undertake similar initiatives.\n* Extensive experimental results based on several state-of-the-art models are presented.\n* The usefulness of the proposed method is also demonstrated in comparison with LLM.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}]

Author rebuttal:

Meta-review: The reviewers were overall positive and did not highlight any critical flaws in the dataset. On that basis I am recommending acceptance.
Decision: Accept (Poster)