SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Image-related Questions related to image modality can be defined as inquiries requiring pixel-level information from CXR images retrieved from EHR, which can aid in analyzing visual diagnoses for individual or cohort patient conditions in real-world medical scenarios. To cater to these queries, we used the 48 MIMIC-CXR-VQA templates (e.g., "List all diseases.") and integrated with expressions to specify our target images (e.g., "The last study of patient 42"). This integration (e.g., "Given the last study of patient 42, list all diseases.") enables retrieval of CXR images from the EHR and subsequent analysis based on natural language requests. We further enhanced the templates focusing on a single patient to include queries that compare two consecutive CXR studies (e.g., "Given the last study of patient 42, are there any newly detected diseases compared to the previous study?") or multiple studies (e.g., "Has patient 42 had any chest X-ray study indicating any anatomical findings in 2023?") from the same patient. This process resulted in 168 templates for the image modality.
Table-related The table modality, a significant part of EHRs, covers questions primarily requiring structured information from EHR tables. These questions relate to patient demographics, diagnoses, procedures, medications, and other clinical details typically recorded in structured EHR formats. EHRSQL, which offers a wealth of questions seeking information from EHR tables, proves to be an invaluable resource in this context. Considering the substantial overlap between the MIMIC-III and MIMIC-IV schemas, we leveraged the question templates from EHRSQL's MIMIC-III templates, adapting them appropriately to fit the MIMIC-IV schema with minimal modifications. This process resulted in 174 templates for the table modality.
Image+Table-related In the image+table modality, all templates are designed to require multi-modal information from both CXR images and structured data from EHRs. We leveraged both MIMIC-CXR-VQA and EHRSQL templates to build multi-modal question templates. Since we recognize the essential role of temporal analysis in multi-modal medical events, we designed templates to capture three primary scenarios: 1) Co-occurring table and CXR events. (e.g., "On the same visit, did patient 42 receive nitroglycerin and have a CXR showing any abnormality in the cardiac silhouette?"); 2) A CXR event following a table event. (e.g., "After being prescribed nitroglycerin, did patient 42 have a CXR during the same visit revealing any abnormality in the cardiac silhouette?") 3) A table event following a CXR event. (e.g., "Was patient 42 prescribed nitroglycerin during the same visit after a CXR showed cardiac silhouette abnormalities?"). These templates allow for comprehensive analysis of combined events, the cause-and-effect relationships in CXR diagnosis,
Figure 3: QA data generation process
Table 3: A comparison of EHRXQA with other EHR QA datasets based on the MIMIC database.
Data source	Image	Table	Text	Patient scope	# of tables / DB	# of questions	compositional
Mimic-VQA [23]	MIMIC-CXR	✓	-	-	single	-	297,723	-
MIMIC-CXR-VQA (ours)	MIMIC-CXR, Chest ImaGenome	✓	-	-	single	-	377,726	✓
MIMICSQL [53]	MIMIC-III	-	✓	-	none, single, group	5	10,000	✓
EHRSQL [33]	MIMIC-III, eICU	-	✓	-	none, single, group	13.5	24,411	✓
DrugEHRQA [4]	MIMIC-III	-	✓	✓	single	3	70,381	-
EHRXQA (ours)	MIMIC-IV, MIMIC-CXR, Chest ImaGenome	✓	✓	-	none, single, group	18	46,152	✓
and relevant follow-up measures related to the CXR diagnosis. To eliminate confusion arising from overlapping information between the CXR and diagnoses/procedures tables, we ensure that questions explicitly specify when a 'CXR study' is necessary. This led to 75 templates for the image+table modality, enabling simulations across diverse scenarios.
4.1.3 SQL/NeuralSQL Annotation
Standard SQL queries are effective for retrieving structured data from EHRs [53, 33], such as demographic information or lab results stored in tables. However, they are not designed to handle unstructured data, such as CXR images, which also contain valuable patient information. This limitation prevents us from using SQL to retrieve answers for complex, multi-modal questions that span both structured and unstructured data. To overcome this limitation, we adopt NeuralSQL, which is inspired by the Binder approach [15]. NeuralSQL acts as an executable representation, extending SQL's capabilities to process unstructured image data. NeuralSQL utilizes a pretrained neural model to extract features from medical images, turning them into a structured format suitable for SQL queries. For more details about our NeuralSQL-based strategy, please refer to Sec. 5.
For Table-related question templates, we utilize the SQL annotations provided by EHRSQL and modify them to be compatible with the MIMIC-IV schema. For question templates related to Image or Image+Table, we annotate them using NeuralSQL representation. The entire SQL/NeuralSQL annotation process was manually undertaken by four graduate students over a span of two months, involving iterative revisions. During this process, the students transformed question templates into their corresponding SQL or NeuralSQL formats.
4.1.4 Data Generation
The question generation process, illustrated in Figure 3, begins with choosing a template at Stage 0, followed by a four-step systematic process (Stages 1-4) that specifies semantics of the template. These steps involve the sampling of visual value (Stage 1), operation value (Stage 2), time template (Stage 3), and condition value (Stage 4). In Stage 1, we augment the question with visual values by filling in object, attribute, and category slots (described in Sec. 3.2.2), tailored specifically for CXR images. Stage 2 involves sampling operation values (e.g., 20s) from a predefined set of options such as [age_group] = (20s, 30s, 40s, 50s, 60 or above), which are independent of the database schema or records. Stage 3 incorporates time templates, translated into natural language expressions to establish a temporal context within the questions. Lastly, Stage 4 incorporates condition value sampling, filling placeholders such as {gender} and {year} to provide context-specific conditions to the question.
The corresponding SQL/NeuralSQL query also contains these slots, filled with the same values during the question creation process, thereby completing the (Question, SQL/NeuralSQL) pair. These (Question, SQL/NeuralSQL) pairs are only added to the data pool if the sampled SQL/NeuralSQL query yields a valid answer when executed. To enhance linguistic diversity, we use GPT-4 to paraphrase each question. These paraphrases are then manually reviewed by our team to ensure quality. Further details can be found in the Appendix C.3.
Figure 4: Overview of our NeuralSQL-based Approach.
4.2 Data Statistics and Comparisons with other EHR QA datasets
EHRXQA consists of a total of 46,152 samples including 16,366 image-related samples, 16,529 table-related samples, and 13,257 samples involving both images and tables. Overall Statistics are summarized in Table 2. For a comprehensive breakdown of the dataset's distribution across various modalities and patient scopes, please refer to Appendix C.4.
Table 2: Overall statistics of EHRXQA including the number of samples for each modality.
train	valid	test
image-related QA	12,860	1,838	1,668
table-related QA	12,961	1,852	1,716
image+table-related QA	10,353	1,480	1,424
total # of samples	36,174	5,170	4,808
Table 3 provides a comparison of EHRXQA with other EHR QA
datasets based on the MIMIC database. Compared to other image-based EHR QA datasets (rows 1-2), EHRXQA incorporates information from EHR tables. This allows for more complex queries about images, such as comparing the clinical condition of two specific images. This feature extends beyond the existing VQA scope and aims to maximize EHR data utilization. Compared with table-based EHR QA datasets (rows 3-5), EHRXQA shows the most complex data structure, featuring up to 18 tables per database and a comprehensive range of patients. These features broaden the spectrum of potential questions that can be posed. To the best of our knowledge, EHRXQA is the first attempt to merge image and tabular modalities in medical QA.
5 NeuralSQL with Visual Question Answering
EHRXQA presents three unique challenges for EHR QA systems that handle both image and table modalities: 1) retrieving and analyzing a single image from the database solely based on natural language expressions; 2) handling multiple images, which include comparative queries across multiple studies; 3) and reasoning across multi-modal data over tables and images. To overcome these challenges, we introduce a NeuralSQL-based approach, inspired by the Binder [15] framework. Our approach integrates a large language model (LLM)-based parser with an external VQA API module, effectively handling both structured information and images. As depicted in Figure 4, the NeuralSQL-based approach consists of two stages:
1. NeuralSQL Parsing: Given a database  $D$  and question  $Q$ , the parser model translates the question  $Q$  to an executable NeuralSQL query  $Z$ . Note that for all Image-related and Image+Table-related questions, we annotated the corresponding NeuralSQL query, as discussed in Sec. 4.1.3. This query features a specific VQA API call function (FUNC_VQA), which handles image-related queries by calling an external VQA model. This API function requires two arguments: (1) a subquestion,  $q_{I}$ , which seeks information related to the image, and (2) the relevant image identifier,  $c_{I}$ , linking to the study_id column in TB_CXR.
2. NeuralSQL Execution: This execution stage involves parsing the NeuralSQL query into an abstract syntax tree (AST), guided by the extended grammar. During this process, the interpreter executes the parsed tree in sequence, including any API calls. Upon encountering a VQA API call, the interpreter employs an internal image loader for the corresponding image(s)  $I$  based on  $c_{I}$ . These image(s) are then fed into the VQA model, which infers the information based on the provided question  $q_{I}$  and image(s)  $I$ . The output of the API call is preserved as a column data object, making it compatible with the standard SQL grammar. This allows the NeuralSQL interpreter to execute the program seamlessly and derive the final answer  $A$ .
In this section, we evaluate medical visual question answering methods on our MIMIC-CXR-VQA dataset (Sec. 6.1). Subsequently, we use the best-performing model as an external VQA API for benchmarking our EHRXQA dataset (Sec. 6.2).
6.1 MIMIC-CXR-VQA
Task & Evaluation We define the VQA task as a multi-label classification with 110 distinct answer labels. This includes 36 objects, 68 attributes, and 4 extras (i.e., 'M', 'F', 'AP', 'PA'), as well as 'yes' and 'no' responses.
In MIMIC-CXR-VQA, verify questions (i.e., "Is there  $\mathbb{S}\{$  attribute\} in  $\mathbb{S}\{$  object\}?") test a model's basic perception, while other questions demand a logical combination of corresponding perception abilities. Therefore, both perception dataset. However, unlike logical operations, perfect perception accuracy in CXRs [7], the mance of MIMIC-CXR-VQA is lower than perception accuracy for single-image video we design a reference model as a classifier. We propose the performance of this model to introduce a new metric by comparing this For each object-attribute pair  $(o,a)$ $m_{rel}$  and attribute.  $m_{VQA}$  and  $m_{ref}$  denote the and  $m_{rel}$  is our proposed relative metric. (AUROC) as our measure  $m$  (denote as A model, not only our relative score, but also details on the reference model, please refer
Table 4: Performance of five baselines on MIMIC-CXR-VQA. To ensure a fair comparison, we pre-trained VLP models (indicated by \*) using the same corpus.
Model	Valid set	Test set
Acc	F1 (micro)	Acc	F1 (micro)	AUCrel
Prior (Most) [2]	26.8	0.27	25.4	0.25	-
Prior (Question) [2]	34.3	0.34	32.4	0.32	-
PubMedCLIP [18]	55.1 ± 1.7	0.56 ± 0.02	54.9 ± 1.3	0.54 ± 0.02	0.82 ± 0.09
PubMedCLIP*	56.6 ± 1.9	0.58 ± 0.02	56.5 ± 2.1	0.56 ± 0.02	0.83 ± 0.09
MedViLL* [40]	64.7 ± 0.2	0.69 ± 0.00	63.6 ± 0.1	0.67 ± 0.00	0.98 ± 0.08
M3AE [13]	68.9 ± 0.2	0.73 ± 0.00	68.9 ± 0.3	0.72 ± 0.00	1.02 ± 0.08
M3AE*	70.2 ± 0.1	0.74 ± 0.00	69.2 ± 0.4	0.73 ± 0.00	1.05 ± 0.09
VQA Baselines We evaluate five VQA baselines: two prior models [2], PubMedCLIP [18], MedViLL [40], and  $\mathbf{M}^3\mathrm{AE}$  [13]. Prior (Most) or Prior (Question) returns the most probable answer estimated from the entire training set or the corresponding question. PubMedCLIP, MedViLL, and  $\mathbf{M}^3\mathrm{AE}$  are vision-language pre-training (VLP) models, each leveraging unique pre-training objectives and architectures. To ensure a fair comparison, we pre-trained all models on the same MIMIC-CXR (image, report) pre-training corpus, with those models denoted by an asterisk  $(\ast)$ . For more details, please refer to Appendix E.1.
Results and Findings Table 4 presents the baseline results on MIMIC-CXR-VQA dataset. The model Prior (Question), which depends solely on language, yields an accuracy of around  $30\%$ . This result attests to the reduced language bias in our dataset, emphasizing the importance of multi-modal reasoning. Among the models evaluated,  $\mathbf{M}^3$ AE achieves the best performance, likely due to its more fine-grained pre-training objectives compared to PubMedCLIP and MedViLL.
6.2 EHRXQA
Task We use semantic parsing to bridge natural language and machine-executable language. The Image-related and Image+Table-related QA scopes are formulated as a Text-to-NeuralSQL task, facilitating complex queries across images and tables. The Table-related QA scope, focusing solely on tabular data, is tackled as a Text-to-SQL task.
Evaluation We employ three metrics to assess the effectiveness of the parsing and execution stages described in Sec. 5, as well as the overall performance of the QA system: 1) Logical Form Accuracy  $(Acc_{LF})$  evaluates the performance of the parsing stage  $(Q \to Z)$ . It computes the accuracy by performing an exact match comparison between the logical form of the predicted program  $\hat{Z}$  and that of the ground truth program  $Z$ ; 2) Ground-truth Execution Accuracy  $(Acc_{EX|gt})$  assesses the accuracy of the execution stage  $(Z \to A)$  by comparing the result of the ground truth program  $Z$  with the ground truth answer  $A$ . For Table-related QA in EHRXQA, this metric yields  $100\%$  accuracy. For Image-related QA and Image+Table-related QA, this equates to measuring the VQA performance; 3) Prediction Execution Accuracy  $(Acc_{EX|pred})$  evaluates the accuracy of execution with the predicted program  $\hat{Z}$ , providing an assessment of the overall system performance, including both parsing and execution stages.
Table 5: Comparison of ChatGPT (gpt-3.5-turbo-0613) with  $\mathbf{M}^3\mathrm{AE}$  model on EHRXQA dataset using two different prompting strategies for Image-, Table-, and Image+Table-related QA.
Model	Prompt	Image-related	Table-related	Image+Table-related
AccLF	AccEX|gt	AccEX|pred	AccLF	AccEX|gt	AccEX|pred	AccLF	AccEX|gt	AccEX|pred
ChatGPT + M3AE	Fixed	1.1	49.4	17.4	4.9	100.0	30.0	4.8	68.8	35.7
BM25 (train)	87.3	49.4	48.2	73.0	100.0	92.9	72.5	68.8	65.9
Baselines We build a strong QA baseline by combining ChatGPT [42] and  $\mathbf{M}^3\mathrm{AE}$  [13], which are outperforming models in the fields of semantic parsing (e.g., Text-to-Query) and medical VQA (e.g., MIMIC-CXR-VQA), respectively. For ChatGPT, we conduct in-context learning [9] (ICL) through two different prompt strategies: 1) Fixed: using fixed N-shot (Question, Query) pairs; 2) BM25 (train) [47]: retrieving N relevant (Question, Query) pairs from the training QA dataset for a given question. These retrieved pairs are then used as few-shot examples. Here, we use N as 10. For  $\mathbf{M}^3\mathrm{AE}$ , we first train it on our MIMIC-CXR-VQA and then deploy it as our external VQA API, integrated within NeuralSQL. For more detailed implementations, please refer to Appendix E.3.
Results and Findings Table 5 shows the performance of EHRXQA, with three metrics for each modality-based scope. The first row of the table shows the performance when using a fixed prompt for all questions, while the second row shows the performance when given a different prompt for each question using BM25. As shown in the Table 5, giving relevant few-shot examples using BM25 significantly boosts performance. In the case of Table-related questions, our model achieves  $92.9\%$  AccEX|pred score with  $73.0\%$  AccLF score. However, when it comes to the remaining questions that rely on image information, our model demonstrates a relatively low performance, even though it maintains a high AccLF score. Specifically, for Image-related questions, the AccLF is  $87.3\%$  as compared to the AccEX|pred of  $48.2\%$ . For Image+Table-related questions, the model achieves an AccLF of  $72.5\%$ , while the AccEX|pred is  $65.9\%$ .
Notably, the model's performance at the execution stage  $(Acc_{EX|pred})$  is affected by the number of images that the model (i.e., VQA model) needs to process. For example, in the context of Image-related QA, we observed that the  $Acc_{EX|pred}$  drops to  $39.6\%$  when the model has to process multiple images (i.e., (Image, single, N-image) scope described in Table 1) within a single patient QA scope. The situation worsens in a group QA scope where the model faces the challenge of accurately predicting a large number of image results, leading to an  $Acc_{EX|pred}$  of  $1.7\%$ . This observed trend contributes to the relatively reduced performance for Image-related  $(48.2\%)$  and Image+Table-related questions  $(65.9\%)$ , even when considering the model's peak overall test set performance  $(69.2\%)$  as detailed in Table 4.
This trend also explains the model showing superior performance ( $Acc_{EX|pred}$ ) on Image+Table-related questions (65.9%) than on Image-related questions (48.2%). Given the complex conditions present in Image+Table-related questions, the scope of images becomes more specified. This leads to a lower number of images to process in comparison to Image-related scenarios, resulting in a relatively higher performance for these multi-modal queries. Overall, the huge gap between  $Acc_{LF}$  and  $Acc_{EX|pred}$  suggests visual perception could be a bigger roadblock to AI models being deployed in clinical practice than logical reasoning, and future research should put as much emphasis on perception as complex logical reasoning.
**Limitations** Though we have carefully designed the dataset, several limitations exist: 1) Since our dataset is based on the MIMIC database, it potentially limits its generalizability. 2) Due to the constrained label scope of Chest Imagenome, our dataset lacks the capability to address more detailed visual questions, such as identifying specific tumor sizes from chest X-rays. 3) Unlike EHRSQL, our model does not include unanswerable questions, an aspect that, if addressed, could enhance our model's comprehensiveness and applicability. Future work should aim to address these constraints.
Future Direction Our study signifies a substantial step forward in multi-modal EHR QA systems, but notable potential for refinement remains. Key future directions include: 1) Enlarging the scope of our dataset by enhancing the multi-modal dialogue system [36]; 2) Incorporating mechanisms to address unanswerable questions or ambiguous images, which is crucial for real-world applications [33]; and 3) Broadening our modality by evolving our dataset to support tri-modal question answering [20, 50]. These forward-looking endeavors will leverage our dataset as a valuable resource, laying the groundwork for more comprehensive and practical healthcare solutions.
We are grateful to Jiho Kim, Jiyoung Lee, Youngjune Lee, JongHak Moon, Hyunseung Chung, and Seungho Kim for their fruitful comments and inspiration. We would like to thank three anonymous reviewers for their time and insightful comments. This work was (partially) supported by Microsoft Research Asia, Institute of Information & Communications Technology Planning & Evaluation (IITP) grant (No.2019-0-00075, RS-2022-00155958), National Research Foundation of Korea (NRF) grant (NRF-2020H1D3A2A03100945), and the Korea Health Industry Development Institute (KHIDI) grant (No.HR21C0198), funded by the Korea government (MSIT, MOHW).
[52] Matthias Urban and Carsten Binnig. Towards multi-modal dbmss for seamless querying of texts and tables. arXiv preprint arXiv:2304.13559, 2023.
[53] Ping Wang, Tian Shi, and Chandan K Reddy. Text-to-sql generation for question answering on electronic medical records. In Proceedings of The Web Conference 2020, pages 350-361, 2020.
[54] Joy T Wu, Nkechinyere Nneka Agu, Ismini Lourentzou, Arjun Sharma, Joseph Alexander Paguio, Jasper Seth Yao, Edward Christopher Dee, William G Mitchell, Satyananda Kashyap, Andrea Giovannini, et al. Chest imagenome dataset for clinical reasoning. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2).
[55] Tao Yu, Rui Zhang, Kai Yang, Michihiro Yasunaga, Dongxu Wang, Zifan Li, James Ma, Irene Li, Qingning Yao, Shanelle Roman, et al. Spider: A large-scale human-labeled dataset for complex and cross-domain semantic parsing and text-to-sql task. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 3911-3921, 2018.
[56] Yilun Zhao, Yunxiang Li, Chenying Li, and Rui Zhang. Multihiertt: Numerical reasoning over multi hierarchical tabular and textual data. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 6588-6600, 2022.
[57] Victor Zhong, Caiming Xiong, and Richard Socher. Seq2sql: Generating structured queries from natural language using reinforcement learning. arXiv preprint arXiv:1709.00103, 2017.