SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Seongsu Bae $^{1*}$ , Daeun Kyung $^{1*}$ , Jaehee Ryu $^{1}$ , Eunbyeol Cho $^{1}$ , Gyubok Lee $^{1}$ , Sunjun Kweon $^{1}$ , Jeongwoo Oh $^{1}$ , Lei Ji $^{2}$ , Eric I-Chao Chang $^{3}$ , Tackeun Kim $^{4}$ , Edward Choi $^{1,\dagger}$ , KAIST $^{1}$  Microsoft Research Asia $^{2}$  Centre of Perceptual and Interactive Intelligence $^{3}$  Seoul National University Bundang Hospital $^{4}$  {seongsu, kyungdaeun, edwardchoi}@kaist.ac.kr $^{1}$
Electronic Health Records (EHRs), which contain patients' medical histories in various multi-modal formats, often overlook the potential for joint reasoning across imaging and table modalities underexplored in current EHR Question Answering (QA) systems. In this paper, we introduce EHRXQA, a novel multi-modal question answering dataset combining structured EHRs and chest X-ray images. To develop our dataset, we first construct two uni-modal resources: 1) The MIMIC-CXR-VQA dataset, our newly created medical visual question answering (VQA) benchmark, specifically designed to augment the imaging modality in EHR QA, and 2) EHRSQL (MIMIC-IV), a refashioned version of a previously established table-based EHR QA dataset. By integrating these two uni-modal resources, we successfully construct a multi-modal EHR QA dataset that necessitates both uni-modal and cross-modal reasoning. To address the unique challenges of multi-modal questions within EHRs, we propose a NeuralSQL-based strategy equipped with an external VQA API. This pioneering endeavor enhances engagement with multi-modal EHR sources and we believe that our dataset can catalyze advances in real-world medical scenarios such as clinical decision-making and research. EHRXQA is available at https://github.com/baeseongsu/ehrxqa.
Electronic Health Records (EHRs) are large-scale databases that store the entire medical history of patients, including but not limited to structured medical records (e.g., diagnosis, procedure, medication), medical images (e.g., chest X-ray, MRI, CT), and clinical text (e.g., discharge summary, nursing note). This wealth of patient information reveals tremendous clinical knowledge about individual patients and cohorts, marking them as an indispensable resource for healthcare professionals (e.g., physicians, nurses, administrators) in routine clinical practice.
Recent years have seen an upsurge in research [33, 34, 44, 46, 53] into question answering (QA) systems for EHRs. These systems are designed to effectively retrieve information from EHRs, each specializing in a different information modality within the records. For instance, table-based EHR QA systems can easily retrieve specific information from structured databases and answer questions like "Did patient 42 undergo a left heart cardiac catheterization procedure in the last hospital visit?" (see EHRSQL part in Figure 1) by executing an SQL query on the relational database. On the other hand, image-based EHR QA (i.e., medical visual question answering) models are designed to handle questions related to individual medical images. For instance, given a question such as "List all
*These authors contributed equally
Figure 1: Our EHRXQA dataset is constructed from three uni-modal resources: MIMIC-IV for the table modality, MIMIC-CXR for the image modality, and Chest ImaGenome as a high-quality annotated version of MIMIC-CXR (not shown in the figure). Our dataset features questions for individual EHR modalities and those requiring multi-modal reasoning. It encompasses three types of QA scope: Image-, Table-, and Image+Table-related QA.
common abnormalities in both the left lung and cardiac silhouette." (see MIMIC-CXR-VQA part in Figure 1) along with a patient's chest radiograph, these models generate a response, thereby serving as an effective aid for radiologists. However, despite their undeniable utility, a main challenge in the current landscape of EHR QA systems lies in their focus on a single information modality, overlooking EHRs' inherently multi-modal nature. To fully utilize EHRs' potential, it is crucial to develop QA systems capable of seamlessly navigating across these multiple modalities such as "Did patient 42 undergo the left heart cardiac catheterization procedure during the last hospital visit, after the chest X-ray revealed any abnormality in the cardiac silhouette within the same period?" (see EHRXQA part in Figure 1). This capability significantly enhances our ability to build a comprehensive model of a patient's status, thereby improving the quality of the clinical decision-making process.
The progression from uni-modal to multi-modal EHR QA is a promising and self-evident step in the healthcare domain. Currently, however, only one multi-modal EHR QA dataset [4] integrates structured EHRs with clinical text. On the other hand, the integration of table modalities with imaging modalities, such as chest X-rays (CXR), remains unexplored [37]. Our research aims to bridge this gap. This has the potential to unlock significant clinical benefits, enhance cross-modal analysis, and catalyze advances in medical research.
To sum up, our contributions are threefold:
- To address the lack of publicly accessible image-based EHR QA datasets that can be combined with structured EHRs, we present MIMIC-CXR-VQA (Sec. 3.2.2). This is a complex, diverse, and large-scale visual question answering dataset in the medical domain. We not only use its questions as a basis for multi-modal EHR questions, but also exploit this dataset to benchmark existing medical VQA approaches.
- We present EHRXQA (Sec. 4), the first multi-modal EHR QA dataset for table and image modality. By leveraging uni-modal resources (i.e., data sources & question templates), we integrate patients' structured databases with their aligned chest X-ray images, thereby creating a comprehensible set of QA pairs covering Image-related, Table-related, Image+Table-related questions.
- We propose a NeuralSQL-based approach (Sec. 5) that integrates Large Language Models (LLMs) with an external VQA application programming interface (API) to handle multi-modal questions over a structured database with images. Despite facing unique challenges of reasoning based on single or multiple images, or even a combination of images and tables, our approach effectively extracts relevant information from multi-modal EHRs in response to natural language queries.
Image-based EHR Question Answering Image-based EHR QA [23-25, 31] is a distinct subset of medical visual question answering (VQA) [1, 5, 21, 22, 32, 38], given that it focuses on answering questions related to a specific patient's single medical image, primarily within the radiography domain. Despite intriguing research directions in existing datasets such as patient-centric QA [24] or dialogue [31], there remains a noticeable gap in efforts to view patient images as an integral part of the EHR database or to synchronize them effectively with the structured tabular data in EHRs.
Presently, MIMIC-CXR [28] is the only publicly available imaging resource that links to patient IDs (i.e., subject IDs) in the MIMIC-IV database [27], offering a comprehensive perspective on EHRs. Although there exist two medical VQA datasets [23, 31] based on MIMIC-CXR, neither is publicly available. Moreover, their question templates are less complex (i.e., they lack complex set operations or logical operations) and are largely encompassed by our question template scope. $^3$
Table-based EHR Question Answering Table-based EHR QA [3, 17, 33, 46, 49, 51, 53] focuses on extracting structured information from a hospital's relational database. The task is typically approached through semantic parsing [6], where natural language utterances are translated into either a query language [35, 55, 57] or domain-specific logical forms [46, 49]. Wang et al. [53] introduced MIMICSQL dataset for the text-to-SQL generation task on MIMIC-III, employing slot-filling for predefined templates and using crowd-sourced paraphrasing. Pampari et al. [44] constructed emrKBQA dataset, a large-scale text-to-logical form dataset tailored for patient-specific QA on MIMIC-III, drawing from the logical forms identified in emrQA [44]. Recently, Lee et al. [33] introduced a novel text-to-SQL dataset, EHRSQL, associated with both MIMIC-III and eICU [45]. This dataset presents unique challenges, including time-sensitive questions and unanswerable queries.
Question Answering over Multi-Modal Knowledge Sources Recent research [10, 12, 14, 16, 39, 48, 50, 52, 56] has delved into generating responses to queries using multi-modal knowledge sources. However, the major challenge when dealing with multi-modal databases, such as EHRs [4], is integrating rich unstructured data (e.g., image, text) into a structured database (e.g., table) and effectively leveraging this information within the QA system. Urban et al. [52] introduced MMDBs, a new category of database systems, which allow seamless querying of text and tables using SQL. Similarly, Chen et al. [14] proposed Symphony, a QA system for multi-modal data lakes, particularly designed to handle text and tables by using a unified representation for multi-modal datasets. Drawing inspiration from recent studies like Binder [15], a training-free neural-symbolic framework that uses GPT-3 Codex [11] to map task inputs to programs, our research broadens the SQL syntax to create a QA system specifically intended for image processing within the database.
3 Preliminary: Ingredients for Multi-Modal EHR QA
3.1 Uni-Modal Data Resources
To construct a comprehensive EHR database that integrates both table and image modalities, we need uni-modal resources that meet our criteria: (i) publicly accessible; (ii) presence of common patients across datasets; (iii) contain high-quality image annotations. After careful consideration, we strategically select three datasets: MIMIC-IV [27] for table modality, MIMIC-CXR [28] for image modality, and Chest ImaGenome [54] as a high-quality annotated version of MIMIC-CXR. Note that all datasets share a significant number of patient IDs (19,264), while incompatible patient IDs exist due to the varying data collection periods. We briefly introduce each of the source datasets. $^{4}$
- MIMIC-IV (v2.2) [27] is a large, freely accessible relational database of deidentified health-related data (e.g., diagnoses, procedures, and treatments) associated with 50,920 patients who stayed in critical care units of Beth Israel Deaconess Medical Center (BIDMC) between 2008-2019.
- MIMIC-CXR (v2.0.0) [28] is a large-scale publicly available dataset of 377,110 chest radiographs associated with 227,827 imaging studies sourced from the BIDMC between 2011-2016. MIMIC-CXR can be linked to MIMIC-IV using lookup tables that connect patient identifiers.
- Chest ImaGenome (v1.0.0) [54], organized with scene graphs for 242,072 frontal images sourced from MIMIC-CXR, illustrates the relationships between anatomical locations and their corresponding attributes within each image. This dataset comprises two primary subsets: the silver $^{5}$  dataset with automatically generated scene graphs for each chest X-ray image, and the gold dataset containing a subset that has been manually validated and corrected by clinicians, serving as a reliable held-out set for research derived from 500 unique patients.
<sup>3</sup>In comparison: Hu et al. [23] provides around 15 templates across 6 types, Kovaleva et al. [31] offers 1 template across 1 type, while our dataset presents 48 templates across 7 types.
4All three datasets are publicly accessible through the PhysioNet platform (https://physionet.org/), with users required to request and obtain credentialed access under its established procedure.
For the silver dataset, given the high inter-annotator agreement score (0.984 for 500 reports) [54], the reliability is strongly suggested. This score substantiates the decision to use the silver dataset for building our MIMIC-CXR-VQA and EHRXQA, providing confidence in the accuracy and quality of the derived information.
3.2 Uni-Modal EHR QA datasets
We aim to build a multi-modal EHR QA dataset featuring questions for each modality individually, as well as those that require cross-modal reasoning. To achieve this, we utilize uni-modal QA datasets based on MIMIC nature. For table modality, we take the existing questions templates from EHRSQL [33], and adapt them to MIMIC-IV. For image modality, to address the lack of diverse question templates and the absence of accessible VQA datasets based on MIMIC-CXR, we craft our templates and further construct a medical VQA dataset called MIMIC-CXR-VQA (Sec. 3.2.2).
3.2.1 Table-based EHR QA: EHRSQL
EHRSQL [33] is a text-to-SQL dataset curated for structured EHRs, assembled from the responses of various hospital staff. EHRSQL provides (Question, SQL) samples for two publicly accessible EHR datasets, namely MIMIC-III [29] and eICU [45], and samples consist of both answerable and unanswerable questions. Since our research scope primarily focuses on building a multi-modal QA dataset, we have selected only the answerable question templates from EHRSQL for MIMIC-III. These templates were converted to align with our MIMIC-IV setting, while maintaining their comprehensive template schema, including multiple value slots (e.g., operation and condition value slots) and time filter slots. For more details about the conversion process of question templates from MIMIC-III to MIMIC-IV, please refer to Appendix B.2.1.
3.2.2 Image-based EHR QA: MIMIC-CXR-VQA
Data Preprocessing We use MIMIC-CXR [28] as our image source and Chest ImaGenome [54] for label information. In MIMIC-CXR, each patient can have multiple studies arranged in chronological order, and each study can contain multiple CXR images. From each study, we select one representative frontal view (i.e., AP, PA) image. We then assign labels to these images derived from the Chest ImaGenome silver/gold datasets. As a result, each CXR image features 563 distinct relations among 36 objects, each linked to several attributes from a pool of 68 attributes (across 5 categories $^{6}$ ). As illustrated in Figure 2, each relation indicates the presence (1) or absence (0) of an attribute (e.g., lung cancer) within a category (e.g., disease), linked to an object (e.g., left lung). For data splitting, we use the machine-generated silver label dataset for training and validation, with a 95:5 split, while the human-labeled gold dataset serves as the testing dataset. For more details of data preprocessing, please refer to Appendix B.2.2.
Figure 2: Upper: Scene graphs of multiple CXR studies derived from the Chest ImaGenome. Lower: Our processed CXR features, obtained from these scene graphs. Due to spatial constraints, only a subset of the original Chest ImaGenome labels is displayed.
CXR Features
"510032"
index	object (36)	attribute (68)	category (5)	relation
1	left lung	lung opacity	anatomical finding	0
2	left lung	pneumonia	disease	0
...	...	...	...	...
42	right lung	lung opacity	anatomical finding	1
43	right lung	pneumonia	disease	1
...	...	...	...	...
563	cardiac silhuoette	cabg grafts	device	1
Question Template Construction We started by analyzing existing medical VQA datasets [1, 5, 21, 22, 32, 38]
and templatized their questions to match our preprocessed data schema (i.e., object, attribute, category), thus handcrafting our initial seed templates. We drew inspiration from general VQA datasets [2, 19, 26, 30], enhancing these seed templates using logical and set operations to create a more diverse and complex set of question templates. We further incorporated clinically relevant factors [32] into our templates, such as the patient's gender, CXR view position, and size-related features (i.e., width ratio between two anatomical locations). As a result, we defined a total of 48 templates, all of which were evaluated by a medical expert for clinical importance. For more details about template construction including a list of our templates, please refer to Appendix B.2.2.
VQA dataset generation We generated our VQA dataset by sampling (image  $I$ , question  $Q$ , answer  $A$ ) triples. For example, consider the template "Is there  $\$\{attribute\}$  in the  $\$\{object\}?$ ". We filled this template using sampled arguments (e.g.,  $\$\{object\} = 'left lung'$ ,  $\$\{attribute\} = 'lung cancer'$ ), which led to the creation of the question  $Q$ : "Is there lung cancer in the left lung?" Next, we sampled an image
6The 5 categories include 'anatomical finding', 'disease', 'device', 'tubes/lines', 'technical assessment'.
Table 1: Sample questions in EHRXQA, categorized by modality-based (Image, Table, Image+Table) and patient-based scope (none, single, group), illustrating our dataset's diversity and complexity.
modality-based	patient-based	Sample question
Image	single	1-image	Given the last study of patient 15439, which anatomical finding is associated with the right lower lung zone, pneumothorax or vascular redistribution?
2-image	Enumerate all diseases that are newly detected based on the last study of patient 19290 in 2103 compared to the previous study.
N-image	How many times has the chest X-ray of patient 18489 shown linear/patchy atelectasis in the left lung on the current hospital visit?
group	Count the number of patients whose chest X-ray studies this year showed any abnormalities in the mediastinum.
Table	none	What's the cost of a drug named lopinavir-ritonavir?
single	Did patient 16164 receive any magnesium lab tests last year?
group	What was the top three diagnosis that had the highest two year mortality rate?
Image+Table	single	Did a chest X-ray study for patient 15110 reveal any anatomical findings within 2 month after the prescription of hydralazine since 2102?
group	Provide the ids of patients in the 20s whose chest X-ray showed low lung volumes in the right lung this month.
$I$  and executed a predefined program $^{7}$  to generate an answer  $A$ . To enrich linguistic diversity while preserving focus on the medical domain [41], we devised a paraphrasing strategy (an average of 16.5 paraphrases for each template) using carefully designed prompts based on GPT-4 [43]. Finally, we present MIMIC-CXR-VQA, a dataset composed of 377,391 unique  $(I, Q, A)$  triples across seven content types $^{8}$ . For a deeper dive into the statistics of MIMIC-CXR-VQA and its comparisons to other medical VQA datasets, please refer to Appendix B.2.2.
4 EHRXQA: A Multi-Modal EHR Question Answering Dataset
4.1 Dataset Construction
In this section, we outline the construction process for the EHRXQA dataset. We begin by integrating CXR images from MIMIC-CXR and tables from MIMIC-IV into our EHRXQA database (see Sec. 4.1.1). Next, we detail the creation of question templates (see Sec. 4.1.2), and the incorporation of the corresponding SQL/NeuralSQL annotations (see Sec. 4.1.3). Finally, we discuss our systematic data generation process (see Sec. 4.1.4) employed to build our EHRXQA dataset.
4.1.1 Database Construction
CXR Integration into MIMIC-IV To cross-reference CXR images with structured EHRs (e.g., to find CXR images of patients who have been prescribed a specific drug), an integrated database system is crucial. To achieve this, we developed an image reference table named TB_CXR. This table comprises six columns: subject_id, hadm_id, study_id, image_id, studydatetime, and viewposition, connecting patient-related identifiers with CXR images of MIMIC-CXR. Through this table, patient CXR images can be retrieved alongside other table data (e.g., diagnosis, procedure, and prescriptions) from MIMIC-IV using the subject_id or hadm_id. For more details on the database construction process, please refer to Appendix C.1.
Timeframe Adjustment We condensed the event times in each patient's records, which originally spanned from 2100 to 2200 due to the de-identification process in MIMIC-IV [27], to a more realistic timeframe (2100-2105). This adjustment was performed while preserving the integrity of CXR images and individual medical event timelines. To enable relative time expressions like 'last year', we set '2105-12-31 23:59:00' as the current time and excluded any records beyond this point. We consider patients without hospital discharge times, due to this exclusion, as currently admitted.
Building Silver/Gold Databases The Chest ImaGenome [54] dataset includes two types of cohorts based on image information: silver (i.e., machine-generated) and gold (i.e., human-labeled). We selected subsets of patients from each cohort to create two distinct databases: the silver database, comprising 800 patients, and the gold database, comprising 400 patients. These databases are utilized for different purposes: the silver database is used for training and validating the QA dataset, while the gold database is used for testing the QA dataset.
For each template, we define a program to produce an answer  $A$  using the given question  $Q$  and relationship information from the preprocessed data (see Sec. 3.2.2) of the image  $I$ .
Questions are divided into 7 categories based on the content of the question: 'presence', 'anatomy', 'attribute', 'abnormality', 'size', 'plane', 'gender'.
4.1.2 Question Template Construction
We define the scope of our question templates using two key criteria: modality-based and patient-based scopes. The modality-based scope classifies templates into three categories, Image-related, Table-related, and Image+Table-related, depending on the type of data modality they require. The patient-based scope classifies templates according to whether they relate to a single patient, a group of patients, or none (i.e., do not relate to specific patients). To accommodate these scopes with diverse and comprehensive question templates, we employ existing uni-modal question resources discussed in Sec. 3.2: MIMIC-CXR-VQA for image modality and EHRSQL for table modality. Examples of our modality- and patient-based question templates, which illustrate the diversity and complexity of EHRXQA dataset, can be found in Table 1.
Recognizing the critical role of time expressions in real-world questions in the hospital workplace [33], we further refined our question templates. We adopted the time filter concept from EHRSQL and applied it to all question templates. This enhancement allows our question templates to better meet the specific needs in clinical practice. Note that these time filters can be categorized into three types: 1) [time_filter_global] restricts the time range of interest, such as 'last year' or 'in 2022'; 2) [time_filter_within], incorporating the keyword 'within', pinpoints events happening within specific temporal boundaries, such as 'within the same hospital visit' or 'within the same day'; 3) [time_filter Exactly] refers to a precise temporal point, such as the 'last CXR study' or a specific date and time like '2105-12-26 15:00:00'.
Our template construction process included 1) clinical needs across both image and table modalities via consulting a medical expert, 2) grounding our templates in these needs for both CXR images and EHR tables, and 3) ensuring clinical relevance. Note that the entire process of designing templates was validated by a board-certified medical expert from the department of neurosurgery to ensure clinical utility. For a full list or an in-depth discussion on template construction strategy, please refer to Appendix C.2. The following details how we tailored question templates for each modality.