SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Shih-Cheng Huang*
mschuang@stanford.edu
Zepeng Huo*
zphuo@stanford.edu
Ethan Steinberg*
ethanid@stanford.edu
Chia-Chun Chiang
chiang.chia-chun@mayo.edu
Matthew P. Lungren
mlungren@microsoft.com
Curtis P. Langlotz
langlotz@stanford.edu
Serena Yeung
syyeung@stanford.edu
Nigam H. Shah
Jason A. Fries
Synthesizing information from multiple data sources plays a crucial role in the practice of modern medicine. Current applications of artificial intelligence in medicine often focus on single-modality data due to a lack of publicly available, multimodal medical datasets. To address this limitation, we introduce INSPECT, which contains de-identified longitudinal records from a large cohort of patients at risk for pulmonary embolism (PE), along with ground truth labels for multiple outcomes. INSPECT contains data from 19,402 patients, including CT images, radiology report impression sections, and structured electronic health record (EHR) data (i.e. demographics, diagnoses, procedures, vitals, and medications). Using INSPECT, we develop and release a benchmark for evaluating several baseline modeling approaches on a variety of important PE related tasks. We evaluate image-only, EHR-only, and multimodal fusion models. Trained models and the de-identified dataset are made available for non-commercial use under a data use agreement. To the best of our knowledge, INSPECT is the largest multimodal dataset integrating 3D medical imaging and EHR for reproducible methods evaluation and research<sup>1</sup>.
The practice of modern medicine is inherently multimodal, where synthesizing information from multiple data sources is essential for diagnosis (identifying which condition a patient currently has) and prognosis (predicting the likely course or outcome of a disease). Physicians routinely analyze patients' current symptoms and past medical history by examining imaging modalities such as X-rays and reviewing electronic health record (EHR) data to diagnose conditions, monitor disease progression, and tailor treatment plans [40, 10]. Artificial intelligence (AI) models capable of emulating the multimodal approach of contemporary medicine hold significant promise for enhancing the efficiency and accuracy of medical diagnosis, prognosis, and treatment planning.
<https://inspect.stanford.edu>
Figure 1: The INSPECT dataset comprises 19,402 patients' structured longitudinal EHRs, which includes diagnosis/procedure codes, labs, medications, vitals, and demographics, as well as 23,248 CT-scans paired with their corresponding radiology report impression section. We curated PE diagnostic and prognostic labels based on these radiology reports and subsequent visit data.
Recent advancements in multimodal fusion strategies have enabled medical AI models to process a wide variety of input modalities, including complimentary imaging [49, 6], EHRs [24], clinical reports [65], genomics [53], and improve model performance. Many prior works have applied multimodal fusion in medical imaging [25, 20, 14], but prognostic tasks have garnered less attention compared to diagnostic ones, primarily due to challenges in obtaining longitudinal data. For instance, prognostic tasks such as predicting 6-month mortality require future data to assign labels and involve inherently longer time horizons than diagnostic tasks. Existing medical imaging datasets are small in size [67], do not include diverse data modalities [44], or have few diagnosis/prognosis labels [38]. Addressing these limitations via new multimodal imaging datasets is essential to further advance AI-driven diagnostic and prognostic tools.
In response to these challenges, we present INSPECT (Integrating Numerous Sources for Prognostic Evaluation of Clinical Timelines), a multimodal dataset of patients at risk for pulmonary embolism (PE). In clinical settings, multimodal data are vital for identifying long-term complications of PE. Specifically, imaging markers from CT pulmonary angiograms (CTPA) improve prediction accuracy for adverse events [52] and combining CTPA data with EHR data increases the effectiveness of automated PE diagnosis [25]. However, the potential benefits of multimodal methods for prognosis in PE have not been fully explored, mainly due to the lack of extensive multimodal datasets with outcome labels. With INSPECT, we aim to aid in the development of new methods for multimodal fusion, tapping into both known and yet-to-be-discovered biomarkers for PE outcome prediction.
INSPECT is made available under a Data Use Agreement (DUA) for non-commercial research use. Our contributions are summarized as follows:
1. A large-scale, multimodal medical dataset. INSPECT contains 23,248 CTPA studies from 19,402 patients, each including: (1) high-resolution CT images (3D volumetric pixel data) with (2) paired radiology report impression sections, (3) structured longitudinal electronic health record (EHR) data and (4) clinically relevant labels, including diagnostic and prognostic labels. Each patient's longitudinal EHR provides a timeline of medical codes, demographics, medications, labs, vitals, procedures, and diagnoses. To our knowledge, this is the first dataset that combines 3D medical imaging with both radiology reports and longitudinal EHRs.
2. A benchmark for PE diagnosis and prognosis. We establish a benchmark for diagnosing and forecasting outcomes of pulmonary embolism through eight clinically important tasks. We assess various imaging and EHR modeling techniques, including individual models using only medical images or EHR data and combined models that use both modalities. All software and trained models used in our benchmark are available open source.
Dataset	Modalities	Counts	Curated Task Labels
Images	Reports	EHR	Patients	Studies	Diagnostic	Prognostic
UK Biobank [7]	MRI, DXA, Ultrasound	X	*	100,000	many	X	X
Open-I [16]	Chest X-ray	✓	X	3,955	7,466	X	X
CheXpert [30]	Chest X-ray	X	X	65,240	224,316	14	X
MIMIC-CXR [32]	Chest X-ray	✓	✓	65,379	227,835	14	X
RSPECT [11]	CT	X	X	12,195	12,195	13	X
RadFusion [68]	CT	X	*	1,794	1,837	1	X
INSPECT (Ours)	CT	✓	✓	19,402	23,248	1	3
Table 1: INSPECT vs. existing multimodal medical image datasets (* denotes partial availability).
2.1 Medical AI for Pulmonary Embolism
Pulmonary embolism (PE) is a serious medical condition responsible for nearly 300,000 hospital admissions and approximately 180,000 fatalities each year in the United States [21]. Despite the high mortality rate associated with PE, research suggests that timely identification and commencement of appropriate treatment strategies can markedly lower both morbidity and mortality rates [41, 42]. A definitive PE diagnosis is achieved through computed tomography pulmonary angiography (CTPA) [39]; however, patients diagnosed with PE typically endure more than six days of diagnostic delay, and a quarter of these patients are misdiagnosed during their initial visit [1, 19]. Estimating long-term patient outcomes is a critical factor that can aid hospitals in efficiently allocating resources and developing an optimal patient care plan. Current practice primarily depends on rudimentary metrics such as the Pulmonary Embolism Severity Index (PESI) scoring system [54], which only considers a limited set of clinical variables.
Many studies have investigated the automation of PE detection and patient triage to alleviate the burden on radiologists [23, 25, 62, 48, 27, 58, 28]. Most of these studies do not incorporate EHR data into models, even though these records contain crucial patient history and demographic details that are essential for accurate clinical interpretation of medical images [24]. Moreover, there is a notable lack of models focused on predicting long-term outcomes for PE patients [4], largely due to the lack of publicly accessible datasets containing these labels. Hence, further research and model development that includes long-term outcome prediction and comprehensive patient EHRs have the potential to significantly improve PE detection and management.
2.2 Multimodal Fusion for Medical Image Applications
Incorporating clinical context is critical for accurate diagnostic interpretation of medical images. Limiting a radiologist's access to patient EHR data significantly reduces their diagnostic accuracy. Research has consistently highlighted the importance of clinical history, vitals, and lab data in accurately interpreting medical images [40, 10, 13, 12, 34]. Similarly, medical imaging models that use patient EHR data have improved accuracy and clinical relevance [24, 3, 22]. Many studies have shown the benefits of adding clinical context rather than relying solely on imaging data [64, 36, 66, 43, 26, 59, 47, 56, 29]. Nevertheless, a majority of these studies rely on a restricted subset of clinical features, primarily due to the dearth of large-scale, multimodal medical datasets. Consequently, these studies are unable to take full advantage of multimodal data fusion, highlighting the need for more comprehensive research approaches and dataset collection efforts.
2.3 Multimodal Datasets
Publicly available medical datasets continue to drive significant advances in medical AI research [30, 33, 68, 11]. However, very few currently available datasets include multiple modalities, are large scale, and have extensive labeled data, particularly in medical domains leveraging 3D medical images (Table 1). The limitations in data availability primarily stem from the inherent challenges associated with the release of medical data. Publicly sharing patient data requires rigorous review processes to safeguard sensitive patient information from inadvertent exposure. Furthermore, the process of labeling is often a labor-intensive and expensive endeavor. The additional challenge of managing the substantial size of 3D medical data further compounds these issues.
Among previous contributions, the MIMIC dataset [33, 31, 32] stands out as a large-scale work incorporating multiple modalities, with linkages provided with 2D chest x-rays. However, MIMIC lacks 3D medical imaging. The UK Biobank contains a variety of medical imaging modalities (e.g. MRIs, ultrasounds) combined with longitudinal medical record data [8]. However, UK Biobank imaging studies are collected prospectively, creating challenges in studying specific medical conditions, and do include corresponding radiology reports [46]. w
Radfusion [68] combines EHR summary statistics with 3D CT scans. However, Radfusion is a small-scale dataset that does not include longitudinal structured EHR data (i.e., timestamped vitals, labs, procedures, diagnoses, etc.), radiology reports, and outcome labels. Lastly, the RSNA-STR PE CT (RSPECT) dataset [11] contains 12,195 CTPA studies, but provides only a single modality, a single case per patient, and does not include prognosis labels. Our study addresses these gaps by introducing a large-scale dataset extracted from 23,248 PE cases and offers multiple modalities and labels, promising to enrich future research in this space.
3 Cohort Definition & Dataset Composition
Our study, approved by the Stanford Institutional Review Board (Appendix A), identified 155,950 cases involving CT pulmonary angiography at Stanford Medicine (2000-2021) using the STAnford Medicine Research Data Repository (STARR) [9]. Our cohort of CTPA cases was defined through a protocol involving random sampling, data cleaning, and inclusion criteria adherence, resulting in a final cohort of 23,248 CTPA cases for 19,402 distinct patients (see Appendix B for details). For each case, we obtained the DICOM (Digital Imaging and Communications in Medicine) files for the CT scans, the corresponding radiology reports for those scans, and structured EHR data from STARR. Each of these was then processed for analysis and de-identification. We also defined canonical training, validation, and test splits that comprise  $80\%$ ,  $5\%$ , and  $15\%$  of the dataset, respectively. We defined splits based on patient IDs, such that each patient only appears in one split.
Demographics Statistics
Attributes	All	Train	Val	Test
Cases	23,248	18,945	(81.5%)	1,089	(4.7%)	3,214	(13.8%)
Patients	19,402	15,789	(81.4%)	913	(4.7%)	2,700	(13.9%)
Overlapping Studies	RSPECT [11]	579	579	(2.5%)	0	(0.00%)	0	(0.00%)
RadFusion [68]	772	772	(3.3%)	0	(0.00%)	0	(0.00%)
Gender	Female	10,733	8,695	(55.1%)	517	(56.6%)	1,521	(56.3%)
Male	8,666	7,091	(44.9%)	396	(43.4%)	1,179	(43.7%)
Unknown	3	3	(0.00%)	0	(0.00%)	0	(0.00%)
Age	0-18	0	0	(0.0%)	0	(0.0%)	0	(0.0%)
18-39	2,912	2,380	(15.1%)	143	(15.7%)	389	(14.4%)
39-69	9,974	8,135	(51.5%)	465	(50.9%)	1,374	(50.9%)
69-89	5,859	4,740	(30.0%)	268	(29.4%)	851	(31.5%)
>89	657	534	(3.4%)	37	(4.1%)	86	(3.2%)
Race	White	10,704	8,722	(55.2%)	502	(55.0%)	1,480	(54.8%)
Asian	2,976	2,378	(15.1%)	152	(16.6%)	446	(16.5%)
Black	1,103	910	(5.8%)	37	(4.1%)	156	(5.8%)
Native	415	337	(2.1%)	22	(2.4%)	56	(2.1%)
Unknown	4,204	3,442	(21.8%)	200	(21.9%)	562	(20.8%)
Ethnicity	Not Hispanic	15,628	12,709	(80.5%)	729	(79.8%)	2,190	(81.1%)
Hispanic	3,018	2,448	(15.5%)	158	(17.3%)	412	(15.3%)
Unknown	756	632	(4.0%)	26	(2.8%)	98	(3.6%)
Table 2: Demographics statistics of the INSPECT dataset. Demographic percentages are marked in light blue. The prevalence of overlapping cases between RSPECT and Radfusion with INSPECT is also indicated. No training data from RSPECT and Radfusion are included in our validation/test sets.
Table Type	Whole Cohort	Per Patient
# Records	Percentage	Median	Min	Max
Measurement	183,820,762	(81.5 %)	3,783	0	500,368
Drug exposure	17,288,279	(7.67 %)	271	0	118,228
Procedure occurrence	8,614,273	(3.82 %)	190	1	35,926
Condition occurrence	8,320,211	(3.69 %)	148	0	27,480
Visit occurrence	5,865,211	(2.60 %)	126	1	16,336
Visit detail	1,355,691	(0.60 %)	23	0	4,840
Device exposure	88,010	(0.03 %)	1	0	682
Person	87,158	(0.03 %)	4	1	48
Death	4,410	(0.001 %)	0	0	13
Total	225,444,005	(100 %)	5,080	7	741,873
Table 3: Summary statistics of the longitudinal EHR data included in INSPECT.
Figure 2: The cumulative probability distribution of EHR timeline lengths before and after CTPA.
Summary statistics of the demographic characteristics of our final cohort are available in Table 2. A small subset of the studies in our dataset has already been published as part of the RSPECT [11] and Radfusion [68] datasets. To maintain the integrity of our model testing process, we have ensured that no training data from the RSPECT and Radfusion datasets are included in our validation and test set. Similarly, we have also ensured that none of the validation and test data from the RSPECT and Radfusion datasets appear in our training set. This guarantees a clear separation between training and testing data, essential for unbiased model evaluation. We have indicated the prevalence of overlapping cases in Table 2. Based on this cohort, we release the following as the INSPECT dataset:
- CTPA: the imaging slices for the CTPAs in our cohort in DICOM format.
- DICOM Headers: a subset of the DICOM headers from the original DICOM file, including Patient ID, study date, instance order in the series, patient position, pixel spacing, rescale slope, rescale intercept, imaging machine manufacturer, and slice thickness. We made sure that patient ID and study date were anonymized to ensure patient privacy.
- Radiology Report Impressions: the impression section of the corresponding radiologist report for all the CTPAs in our cohort.
- Structured Data From EHRs: de-identified structured data from longitudinal EHR records for each patient in our cohort, including diagnoses, procedures, lab results, medications, and demographics. Each data element consists of a timestamp for when the event occurred, a code signifying the type of event, and optionally, a value (for lab results and vitals). All clinical events and known encounters with Stanford Health Care are included. A distribution showing the event frequency is in Table 3.
A detailed description of the formatting, hosting, and licensing details of INSPECT is in AppendixC.
4 Benchmark
In addition to the INSPECT dataset, we developed a benchmark for evaluating predictive models on our cohort. The code for this benchmark is included in the supplement under an open-source license.
4.1 Data Processing
CTPA Each CTPA exam is preprocessed by extracting the pixel data from the original DICOM format. We linearly transform the extracted pixel data in Hounsfield Units (HU) using the rescale slope and intercept recorded in the DICOM file. Specifically, each image  $x$  is processed with  $x = x*s + b$ , where  $s$  is the rescale slope and  $b$  is the rescale intercept.
Radiologist Reports All CTPA exams are accompanied by a radiology report that contains a summary of the patient's medical history and detailed descriptions of the medical conditions observed by the radiologist. Using a rule-based system, we process these reports by extracting the impression section - a summary of the most important findings and possible causes. In addition, We deidentify the impression section by replacing dates and names with deidentified keywords.
Electronic Health Records Our source EHR records are stored in the OMOP schema [57]. They contain longitudinal EHR for patients seen at Stanford Health Care (comprised of an adult hospital and outpatient clinics) and Stanford Children's Hospital. Each record contains demographic information (age, sex, ethnicity), and coded clinical information (diagnosis codes, lab test orders and results, medication orders, procedures, and visits). The cumulative probability distribution of patient EHR timeline lengths before and after CTPA procedures is shown in Figure 2. We processed the EHR records using the FEMR (https://github.com/som-shahlab/femr) software package and exported data for release in FEMR's CSV format. In order to enable release, we anonymize INSPECT by introducing random time shifts for every patient, removing structured patient identifiers, and removing all unstructured text.
O	T	L	All	Train	Val	Test
PE	N/A	pos. neg.	4,689 18,559	3,924 15,021	(20.7 %) (79.3 %)	188 901	(17.3 %) (82.7 %)	577 2,637	(18.0 %) (82.0 %)
Mort	1 m	pos. neg. cen.	1,200 20,803 1,245	986 16,930 1,029	(5.2 %) (89.4 %) (5.4 %)	54 991 44	(5.0 %) (91.0 %) (4.0 %)	160 2,882 172	(5.0 %) (89.7 %) (5.4 %)
6 m	pos. neg. cen.	2,389 18,552 2,307	1,963 15,075 1,907	(10.4 %) (79.6 %) (10.1 %)	103 900 86	(9.5 %) (82.6 %) (7.9 %)	323 2,577 314	(10.0 %) (80.2 %) (9.8 %)
12 m	pos. neg. cen.	2,916 17,157 3,175	2,390 13,936 2,619	(12.6 %) (73.6 %) (13.8 %)	129 829 131	(11.8 %) (76.1 %) (12.0 %)	397 2,392 425	(12.4 %) (74.4 %) (13.2 %)
Re-ad	1 m	pos. neg. cen.	857 20,774 1,617	695 16,898 1,352	(3.7 %) (89.2 %) (7.1 %)	38 997 54	(3.5 %) (91.6 %) (5.0 %)	124 2,879 211	(3.9 %) (89.6 %) (6.6 %)
6 m	pos. neg. cen.	2,185 17,953 3,110	1,778 14,585 2,582	(9.4 %) (77.0 %) (13.6 %)	99 878 112	(9.1 %) (80.6 %) (10.3 %)	308 2,490 416	(9.6 %) (77.5 %) (12.9 %)
12 m	pos. neg. cen.	2,826 16,253 4,169	2,291 13,201 3,453	(12.1 %) (69.7 %) (18.2 %)	130 794 165	(11.9 %) (72.9 %) (15.2 %)	405 2,258 551	(12.6 %) (70.3 %) (17.1 %)
PH	12 m	pos. neg. cen.	2,726 16,503 4,019	2,242 13,389 3,314	(11.8 %) (70.7 %) (17.5 %)	124 804 161	(11.4 %) (73.8 %) (14.8 %)	360 2,310 544	(11.2 %) (71.9 %) (16.9 %)
Table 4: Task statistics for INSPECT. O: Outcome. T: Time Horizon. L: Label Value, PE: Pulmonary Embolism, Mort: In-Hospital Mortality, Re-ad: Re-admission, PH: Pulmonary Hypertension.
4.2 Task Definitions
Formally, given a set of multi-variate features,  $\mathbf{x}_1,\dots,\mathbf{x}_N$ , which are encoded from the occurrences of the selected covariates in Table 3, which is a composite set of clinical events happening in continuous time steps  $t_1,\ldots ,t_N$ . The goal is to train a model to approach the posterior probability of predicting the future event  $t_{i + m}$  at a specific time horizon:  $p[(t_{i + m},y)|(t_i,\mathbf{x}_i)]$ , where  $\mathbf{x}_i$  is an accumulative feature at time  $t_i$  that encodes information from features from all previous time steps  $\mathbf{x}_i = f(\mathbf{x}_1,\dots,\mathbf{x}_{i - 1})$  and  $f(\cdot)$  is the EHR modality model to be learned. The time horizon  $m$  was a set of predefined periods in the future, and  $y$  was a binary variable to indicate the patient having the medical event at the timestamp  $t_{i + m}$ . When combined with other modalities, e.g. imaging modality features, we augmented the output of EHR model  $y_{i}^{\mathrm{EHR}}$  at prediction time  $t_i$  with a late fusion:
$$
\hat {y} := \mathbf {w} \left[ \begin{array}{c} y ^ {\mathrm {E H R}} \\ \vdots \\ y ^ {\text {i m a g e}} \end{array} \right], \tag {1}
$$
where the fusion weights  $\mathbf{w}$  were learned from the validation set.
In the following sections, we elaborate on the precise definition for each task and how the corresponding labels are generated. Table 4 contains various statistics on the labels for each task.
4.2.1 Diagnostic Tasks
Pulmonary Embolism (PE): We construct a pulmonary embolism diagnostic task that classifies whether pulmonary embolism is diagnosed based on the patient's CT scan. Labels are generated by applying an NLP model to the impression section of the corresponding radiology report. Specifically, we first fine-tune a Clinical Longformer [45] model to predict pulmonary embolism diagnoses given the impression section of a text radiology report. Ground truth labels for the reports are manually collected by [5]. Subsequently, we apply the fine-tuned model to all the impression sections of all studies in our dataset to assign a pulmonary embolism diagnosis (or lack of diagnosis) to every patient in our cohort. Appendix D describes how this model was trained and how we validated its performance.
4.2.2 Prognostic Tasks
For the prognostic tasks, we attempt to predict whether or not a specific event will occur in the future within a specified time horizon for a given patient. To handle missing data, patients without a recorded prognosis event and those lacking data up to the time horizon are considered censored. These patients are excluded from both training and evaluation.
The event definitions are as follows:
- Pulmonary Hypertension (PH): A set of 29 International Classification of Diseases (ICD) codes to identify pulmonary hypertension.
- In-Hospital Mortality: We use the in-hospital mortality events provided by STARR-OMOP.
- Re-admission: We use the inpatient readmission events provided by STARR-OMOP.
Appendix D contains the set of ICD codes used for PH definition and outlines the methodology employed to validate this set of codes.
4.3 Baseline Models
We set up several common modeling approaches for each data type to serve as baselines, including image-only, EHR-only, and multimodal fusion models. The following are brief descriptions of each overall approach. The baseline model construction is shown in Figure 3. Full details, including hyperparameter tuning, can be found in Appendix F.
Figure 3: Baseline Models. We evaluate both single modality models and multi-modal late fusion models that incorporate data from both images and EHRs as baselines. For CT input, we use an LRCN (Long-term Recurrent Convolutional) model, while for structured EHR input, we employ MOTOR and gradient-boosted trees. Our multi-modal fusion baseline utilizes a late fusion approach, learning a weighted mean from each individual modality's predicted probability.
4.3.1 Imaging
We resize all CTPA exams to 256 by 256 pixels. Following this, we refine the focus of each slice through center cropping, resulting in a 224 by 224 pixel matrix. Subsequently, we introduce three viewing windows to highlight particular structures. Each of these windows offers an optimal view for a specific medical perspective: the lung, pulmonary embolism, and mediastinum. We then stack the three view windows into three channels, giving us an array of  $224 \times 224 \times 3$  for each CT slice. Finally, we normalize each CTPA exam to be zero-centered using ImageNet mean and standard deviation.