SlowGuess/NeurIPS_2023_OCR huggingface dataset:
AUROC While the exact match accuracy is a useful metric, it may not fully capture the model's performance since it does not consider partial credits, especially in the case of Query questions that require consideration of much more attributes. To provide a more comprehensive evaluation, we employ the area under ROC curve (AUROC) as another metric. When calculating AUROC, we adopt a cautious approach by only considering the "valid" answer candidates for each question. This approach aims to prevent overestimation, as the model might naturally assign lower scores to "invalid" answer options. For example, in a question like "Which noise does this ECG show, baseline drift or static noise?", we exclusively consider the scores of the answer options baseline drift and static noise. We collect scores for each answer option over all the samples and compute macro-averaged AUROC among the answer options.
4.2 Upper bound experiments
In the field of clinical medicine, even experienced medical practitioners cannot be entirely certain when making crucial decisions, such as diagnosing a patient's condition. Similarly, the ECG-QA dataset can also suffer from this inherent uncertainty even though we extracted attributes from the existing annotations made by expert cardiologists. As  $100\%$  accuracy is unlikely to attain due to the inherent uncertainty, we aim to estimate the upper bound performance a model can achieve with our dataset, and use it as a reference when evaluating the model performance.
Within our dataset, we speculate that questions of the Single-Verify type necessitate basic perceptual abilities while other question types can be solved by logically combining these perceptual abilities. For example, Single-Choose questions can be answered by verifying the presence of each attribute in the given two options, and similarly, Single-Query questions can be solved by verifying the presence of each element within the specified attribute set. Consequently, we hypothesize that achieving high performance on the whole ECG-QA dataset is unlikely without a high level of perceptual ability. Based on this hypothesis, we can estimate the upper bound performances for the whole ECG-QA dataset by measuring the upper bound performances of the Single-Verify samples. To this end, we designed the following experiments.
We convert all the Single QA samples (Single-Verify, Single-Choose, Single-Query) in the training set into the format that ECG classification models can process, and train the classification models using the converted training samples. Similarly, after converting the Single-Verify samples in the test set, we estimate the upper bound performance by measuring performances on the converted Single-Verify test samples. Then, we compare this upper bound with Single-Verify performances of QA models to show how much the QA baselines can be improved in terms of their perceptual ability. The detailed process of converting QA samples into ECG classification format is described in Supplementary C.1.2.
For these experiments, we employ powerful ECG classification models classifying all the individual attributes present in the Single-Verify samples. The models we use include a Transformer-based model pre-trained with the W2V+CMSC+RLM [19] method, Resnet with Attention [18], and SE-
Table 3: Macro-averaged test performances of upper bound models over all attributes for Single-Verify questions
Upper bound Model	Acc.	AUROC
W2V+CMSC+RLM [19]	83.0±0.4	0.864±0.003
Resnet-Attention [18]	82.6±0.3	0.875±0.002
SE-WRN [6]	83.1±0.3	0.883±0.002
MAX	85.4±0.4	0.907±0.002
Table 4: Macro-averaged test performances of QA models over all attributes for Single-Verify questions.
QA Model	Acc.	AUROC
M3AE† [2]	80.8±0.3	0.808±0.006
MedViLL† [17]	79.8±0.3	0.809±0.005
Fusion Transf.	76.4±0.6	0.764±0.010
WRN [6]. In addition, to present the maximized upper bound, we derive another model that takes only the maximum score among the three models for each attribute, which is denoted as MAX. Detailed model implementations and training configurations are presented in Supplementary C.1.2.
4.3 Modeling with LLMs
As for one of the future research directions with our dataset, we further investigated the possibility of leveraging LLMs for ECG-QA. Inspired by ChatCAD [29], for each QA sample, we transformed ECGs into text descriptions using the output from the trained upper bound model (SE-WRN) and forwarded them to several OpenAI's GPT models (gpt-4, gpt-3.5-turbo, text-davinci-003) $^{4}$  along with the corresponding question. Due to the restricted quota of OpenAI's api usage policy, we randomly sampled  $10\%$  from the ECG-QA test set and conducted experiments only once for each LLM model. The detailed processes including prompts that we used are described in Supplementary C.1.3.
QA results The baseline results are presented in Table 2. We also report test performances for different attribute types in Supplementary C.2. As expected, Blind and Deaf Transformer exhibit poor performance while other models all achieve higher scores compared to the prior model (per Q-type majority), indicating that our dataset cannot be solved by solely seeing each question and ECG separately. Furthermore, among the top three models  $(\mathbf{M}^3\mathrm{AE}^\dagger$  , MedViLL, and Fusion Transformer), the pre-trained models  $(\mathbf{M}^{3}\mathbf{A}\mathbf{E}^{\dagger}$  and MedViLL) outperform Fusion Transformer, which demonstrates the potential advantages of utilizing novel multi-modal pre-training methods for our dataset. Additionally, the lower performance of Choose or Query questions compared to Verify questions suggests that the primary challenges in our dataset lie on a model's ability to learn logical and set operations based on the basic perceptual abilities that can be acquired from Verify questions.
Upper bound results The results of the upper bound experiments are reported in Table 3 and 4. When we compare SE-WRN, which is the best upper bound model, with the best QA model,  $\mathrm{M}^3\mathrm{AE}^\dagger$ , we can see that the perceptual ability of baseline models can be improved by  $2.3\% \mathrm{p}$  and  $7.5\% \mathrm{p}$  in terms of EM accuracy and AUROC, respectively. Moreover, when comparing with MAX, which is expected to show a higher upper bound, the differences are increased to  $4.6\% \mathrm{p}$  in EM accuracy and  $9.9\% \mathrm{p}$  in AUROC. We believe these upper bound results can serve as a useful yardstick for assessing the basic perceptual ability required for more complicated questions such as Choose or Query.
LLM modeling results The results of the experiments with LLMs are presented in Table 5. Interestingly, the performance of all the GPT models did not surpass that of the QA baseline model. We speculate that this is due to two primary reasons: 1) the upper bound model (i.e. SE-WRN) fails to accurately extract necessary information, and 2) some questions were too complicated to be answered with a zero-shot prompt. Since LLMs fully rely on the ECG classification model for interpreting the ECGs, their performance inevitably depends on the capabilities of the ECG classification model. However, we cannot guarantee that SE-WRN is such a perfect model that always outputs accurate interpretations, because it has been trained with a limited set of QA training set to measure only the upper bound of the perceptual ability. Therefore, we expect significant performance gains if we have
As all of these experiments were conducted prior to June 7th, 2023, the associated models were referencing the earlier legacy versions. More precisely, at the time of the experiments, gpt-4 corresponded to gpt-4-0314, and gpt-3.5-turbo matched with gpt-3.5-turbo-0301.
Table 5: Test EM accuracies for different question types. Note that we randomly sampled  $10\%$  from the ECG-QA test set for each question type to test the models due to the restricted quota of OpenAI's api usage policy. The best performance for each question type are highlighted with **boldface**.
Question Type	per Q-type majority	SE-WRN + gpt-4	SE-WRN + gpt-3.5-turbo	SE-WRN + text-davinci-003	M3AE† [2]
S-Verify	69.1	71.0	69.3	75.0	76.0±0.7
S-Choose	30.7	48.1	36.1	37.8	58.2±0.9
S-Query	25.3	35.7	31.1	36.0	40.0±1.6
CC-Verify	58.4	54.9	58.2	56.3	74.7±1.2
CC-Query	17.3	13.0	10.5	15.4	21.2±2.0
CI-Verify	67.0	68.8	64.1	71.5	75.2±1.7
CI-Query	1.32	2.53	1.32	1.40	4.36±0.7
S: Single, CC: Comparison-Consecutive, CI: Comparison-Irrelevant
a strong classification model that can extract all the existing information from an ECG, and fine-tune LLMs with our QA training set (i.e., applying instruction learning to LLMs).
In this work, we present ECG-QA, the first QA dataset that incorporates ECG data for question answering tasks. Our dataset is designed to ensure clinical relevance and has been validated by an ECG expert. We created carefully designed question templates, which leverage clinically meaningful attributes extracted from the PTB-XL dataset, to generate a diverse collection of questions, including those that require the comparison of two ECGs. We believe that our dataset has the potential to significantly advance the field of ECG question answering research and contribute to the improvement of clinical practice in analyzing ECG data.
As for the future research directions with ECG-QA, one of the promising avenues is the exploration of multi-modal LLMs that can simultaneously process both ECG signals and natural language. While there is extensive work on LLMs that combine vision and language, there has been limited research on models that integrate signal processing with natural language. We believe our dataset can serve as an excellent testbed for such models.
Despite our best efforts to create the current version of the dataset, there are some limitations as follows.
Small number of ECGs Due to the limited number of ECGs available in the original dataset (PTB-XL), our dataset was constructed using a relatively small number of ECGs ( $\sim 16\mathrm{k}$ ), which leads that questions involving too rare symptoms (e.g., Wolf-Parkinson-White syndrome) could not be included. To provide more diverse combinations of ECGs and questions by incorporating questions regarding very rare attributes, we are planning to employ another dataset that is larger than the PTB-XL dataset such as MIMIC-IV-ECG [5], which is planned to be released in late 2023.
Upper-bound of the dataset As mentioned in Section 4.2, given the intricacies of the medical field, even medical experts cannot provide  $100\%$  accurate diagnoses for all questions. Thus, the upper-bound of the dataset itself is not expected to be  $100\%$ . To address this, we conducted experiments demonstrating the estimated upper-bound for each question type and attribute.
Old version of SCP-ECG standard Despite SCP-ECG v3.0 being the latest version, the metadata of the original dataset, PTB-XL, follows the SCP-ECG v0.4 standard. Consequently, in ECG-QA, we were constrained to categorize various symptoms based on the SCP-ECG v0.4 standard. However, after investigating how the SCP codes in SCP-ECG v3.0 are categorized, we found that there is only a little difference between SCP-ECG v0.4 and v3.0 regarding the SCP codes used in PTB-XL. Among the SCP codes used in PTB-XL, only one SCP code (BIGU, bigeminal pattern - unknown origin,
SV or Ventricular) has a different representation, which has changed to "SVBIG" (supraventricular bigeminy BIGU bigeminal pattern - unknown origin, SV or Ventricular) in SCP-ECG v3.0. The rest of the SCP codes have maintained their codes and definitions intact in SCP-ECG v3.0. Therefore, we believe that the impact of the differences between the two versions will not be significant in the ECG-QA dataset.
Automatic generation of paraphrases Although the paraphrases were manually curated, the initial candidates were automatically generated by ChatGPT, which may not be an optimal strategy. We expect that paraphrases could have been more diverse if we had involved medical practitioners in manually generating paraphrases.
This work was supported by Institute of Information & communications Technology Planning & Evaluation (IITP) grant (No.2019-0-00075), National Research Foundation of Korea (NRF) grant (NRF-2020H1D3A2A03100945), and Korea Medical Device Development Fund grant (Project Number: 1711138160, KMDF_PR_20200901_0097), funded by the Korea government (MSIT, MOTIE, MOHW, MFDS).