SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Figure 3 illustrates the two-step process of our CTPA model, encompassing feature extraction and sequential modeling. First, we employ a pretrained image encoder for feature extraction, processing each CT slice into a latent representation. Subsequently, all extracted features from a given CT series are inputted into a sequence encoder for our final prediction. As a baseline, we leverage a ResNetV2 model pretrained on BigTransfer [37] and finetuned on the RSPECT dataset [11] as slice encoder. For a sequence encoder, we use either an LSTM, GRU, or Transformer model [17].
4.3.2 Structured EHRs
Gradient-boosted Trees
In order to build a gradient-boosted tree model, we first featurize our structured EHR data using a common count-featurization approach [60] that creates a matrix that counts the number of times each code appears in the EHR before the index date. We then train LightGBM [35] models on these count matrix features.
Structured EHR Foundation Model
To address the difficulty of training a deep learning model on small datasets, we adapt MOTOR [63], a foundation model that was pretrained on Stanford structured EHR data. MOTOR was pretrained using time-to-event tasks, making it well-aligned with our desired prognostic prediction tasks. We ensure that none of the patients in our validation or test set were used for pretraining. We then fit a linear probe (i.e., a linear layer and frozen MOTOR backbone) for all tasks.
4.3.3 Multimodal Fusion
We evaluate multimodal fusion strategies[24] to combine our three baseline models. Our fusion models (see Figure 3) aggregate prediction probabilities from individual models by taking a weighted mean of these probabilities of each single-modality model. The weights are trained by learning a logistic regression model on the validation set using individual model probabilities as features.
Input Modality	Diagnostic	Prognostic
Image	EHR	PE	In-Hospital Mortality	Re-admission	PH
CT	M	G	(+)	1 m	6 m	12 m	1 m	6 m	12 m	12 m
✓	0.721	0.794	0.755	0.748	0.549	0.515	0.525	0.661
✓	0.677	0.923	0.901	0.892	0.773	0.779	0.767	0.824
✓	0.681	0.848	0.865	0.855	0.737	0.740	0.728	0.828
✓	✓	0.761	0.924	0.903	0.895	0.774	0.777	0.764	0.820
✓	✓	0.765	0.867	0.875	0.866	0.740	0.736	0.722	0.830
✓	✓	0.699	0.922	0.903	0.892	0.782	0.786	0.774	0.849
✓	✓	✓	0.771	0.924	0.904	0.895	0.782	0.784	0.771	0.843
Table 5: The performance in AUROC for our different baseline modeling strategies, including various late fusions. CT is the CTPA based LRCN (Long-term Recurrent Convolutional Networks) model, M is the structured EHR based MOTOR model, and G is the structured EHR based gradient-boosted trees model. The best overall models are bolded and the best individual models are underlined.
5 Experiments And Results
To validate our dataset and provide some baselines, we perform experiments applying each of our three modeling strategies to each of our tasks. We also perform model fusion on each combination of individual models. Details on the computational resources used to run our experiments are in Appendix G. We also release the trained model weights learned in our experiments as part of our dataset, so our experiments can be reproduced (see Appendix C).
Table 5 contains the model performance in terms of AUROC for each of our approaches on each task. Confidence intervals can be found in Appendix H. When considering individual models, we find that the structured EHR models perform better on prognostic tasks while the CT model performs better on the diagnostic PE task. This is to be expected given that our source PE diagnoses are defined using CT scans, so the CT modality contains the information that most directly solves the diagnostic task.
We find that model fusion between the CT model and the structured EHR models helps improve performance on the diagnostic PE task but does not improve performance on the prognostic tasks. While prior work has identified imaging biomarkers related to chronic disease using CT chest imaging [51, 18], it is unclear if current deep learning models are able to take advantage of these signals. We also note that our image-only model does not match some state-of-the-art models' performance [50] on predicting PE on a similar dataset, i.e. RSPECT [11]. We posit the difference might come from nuanced label definition, where our PE definition (shown in Appendix D) has incorporated subsegmental PE where the RSPECT dataset did not.
Our work to develop INSPECT represents a significant step forward in multimodal, multi-label, multi-timeframe medical AI research, contributing a rich, large-scale dataset and benchmarks in the context of pulmonary embolism. INSPECT encompasses diverse modalities – high-quality CT images, radiology reports, and structured EHRs – enabling the development of benchmark predictive models for PE diagnosis and prognostication. To the best of our knowledge, we are the first to link longitudinal EHR data to 3D medical images and their paired radiology reports. This dataset provides opportunities for the community to derive additional diagnostic/prognostic labels (e.g. diagnosis codes, procedure codes, lab results, medications) for either model pretraining or downstream tasks.
Our preliminary findings suggest that the integration of medical imaging and structured EHRs improves performance in diagnosing PE. However, we also find that incorporating medical imaging for prognostic tasks does not improve predictive performance, especially on the important pulmonary hypertension prediction task. These results conflict with domain knowledge and medical literature, where it has been demonstrated that CT images contain information on some of the causes of pulmonary hypertension [2]. The lack of improved performance in our study suggests that there is untapped potential in our existing techniques, either in the fundamental imaging models or the
synthesis of the imaging model output with models trained using EHR data. By releasing INSPECT, we hope to enable the research community to explore this challenge.
6.1 Societal Implications
The process of releasing comprehensive patient timelines carries the inherent risk of exposing identifiable information. To mitigate this risk, we have adhered to the best practices for data anonymization in accordance with HIPAA compliance standards. Further, we have released all datasets and model weights under DUA, which is a standard procedure for medical data, thus ensuring controlled access to the data and models. However, it is important to note that using these data and/or model weights to provide medical advice and or make care decisions is beyond the intended scope of use for the INSPECT dataset and associated models. To emphasize this, we have specified in our DUA that INSPECT data and models are for research purposes only, not for clinical decision-making.
Firstly, INSPECT only contains data from a single site (Stanford Health Care), and models trained on INSPECT may not generalize to other patient populations. Secondly, labels are assigned based on NLP output and source EHR data, not manual chart review, and thus may be inaccurate in some cases. However, we have taken several steps to mitigate this, as detailed in Appendix D. Finally, for each CTPA image, we release only the impression section of the corresponding radiology report as de-identification protocols preclude releasing the entire note. This limits some analyses and experiments that would require entire radiology reports, beyond the impression section.
There are two main contributions to this work. First, we present a large-scale medical dataset INSPECT with multiple modalities, comprising health records from 19,402 patients, complete with high-quality CT images, portions of accompanying radiology reports, and structured data from patient EHRs. Second, we use this dataset to create a benchmark for a variety of important pulmonary embolism related tasks, with included baseline models. In conclusion, this work has laid the foundation for future research into multimodal fusion strategies for integrating 3D medical imaging data and patient EHR data. By openly sharing INSPECT, we hope to ignite new advances in this critical area of healthcare.
Research reported in this publication was supported by the National Heart, Lung, And Blood Institute of the National Institutes of Health Awards R01HL155410 and R01HL144555, the National Library Of Medicine of the National Institutes of Health Award R01LM012966, and AIMI-HAI seed grant at Stanford University. We would also like to thank the Clinical Excellence Research Center (CERC) at Stanford for their support. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health.
[68] Yuyin Zhou, Shih-Cheng Huang, Jason Alan Fries, Alaa Youssef, Timothy J Amrhein, Marcello Chang, Imon Banerjee, Daniel Rubin, Lei Xing, Nigam Shah, et al. Radfusion: Benchmarking performance and fairness for multimodal pulmonary embolism detection from ct and ehr. arXiv preprint arXiv:2111.11665, 2021.
A IRB Approval and Data De-identification
Release of INSPECT was approved by the Stanford University Institutional Review Board (IRB), given data privacy review via a standardized workflow conducted by the Center for Artificial Intelligence in Medicine and Imaging (AIMI) and the University Privacy Office. Our study was approved by the Stanford University Administrative Panel on Human Subjects Research, protocol #24883, and included a waiver of consent. All included patients from SHC signed a privacy notice, which informs them that their records may be used for research purposes given approval by the IRB.
All INSPECT data (CT scans, DICOM metadata, radiology impression sections, EHR timelines) are manually reviewed by AIMI to confirm any protected health information (PHI) is removed before public release. We de-identify each modality as follows:
EHR Timelines: All dates are anonymized by using per-patient time jittering. We apply the same date transformation procedure used by MIMIC-III, specifically: "[d]ates were shifted into the future by a random offset for each individual patient in a consistent manner to preserve intervals, resulting in stays which occur sometime between the years 2100 and 2200" [33]. We remove all patients  $>89$  years of age. We further remove all unstructured text fields that do not map to controlled vocabularies (e.g., SNOMED, LOINC) to prevent PHI leakage. We use OHDSI Athena [61] ontologies to describe our data, which includes both public ontologies like ICD-10 as well as OHDSI specific ontologies such as Race/Gender. The full list of ontologies used is in Table 6..
OMOP Extension
Medicare Specialty
CPT4
ICD9Proc
RxNorm
SNOMED
RxNorm Extension
Cancer Modifier
ICD10PCS
CMS Place of Service
ICDO3
LOINC
HCPCS
Table 6: OHDSI Athena ontolgies used in our benchmark
CT Scans: Each CT scan slice is manually reviewed for PHI, with slices containing patient information removed from the CT scan.
Radiology Notes: Radiology notes are preprocessed to include only the impression section, i.e., the description of radiologist findings in the corresponding CT scan. Each note is processed to tag names, locations, dates, telephone numbers and other HIPAA protected identifiers such as MRNs and accession numbers. These tags are then replaced with anonymized placeholder text. All deidentified notes are then manually reviewed to remove any additional PHI.
Figure 4: A flowchart of our cohort definition process.
B Cohort Definition
The flowchart of our cohort definition protocol is illustrated in Figure 4. With the approval from Stanford Institutional Review Board's (IRB), we identified 155,590 cases with the CT pulmonary angiography (CTPA) procedure code from the STAnford medicine Research data Repository (STARR) [15]. STARR data contains routinely collected EHR data from Stanford Health Care covering the time period of 2000 to 2021. We mapped each of these studies to their respective CTPA based on the procedure date. In instances where there was no exact match (1,296 cases) we extended the search to 10 days post-procedure date. From the mappings, we sampled uniformly at random 26,475 CT scans (chosen due to file storage constraints) and their corresponding radiology report.
The data cleaning phase followed, where we removed cases without a report or an impression section. This refinement process resulted in 24,474 cases for further analysis. We then selected the most relevant CTPA series per study by enforcing a slice thickness constraint between  $1.0\mathrm{mm}$  and  $3.0\mathrm{mm}$ , favoring thicker slice series. Additionally, CTs with over 600 or under 50 slices were removed from consideration. This filtering resulted in 24,003 studies. We further eliminated the test and validation data from RSNA and Radfusion in our training split and the training data thereof in our test and validation split. The patients who are minors (age  $< 18$  years old) are removed due to privacy policy. The final selection phase involved eliminating cases with corrupted DICOMs and cases from the same patient that mapped to an identical study. The final result is a collection of 23,248 CTPA studies from a set of 19,402 unique patients.
C Dataset Documentation
C.1 Hosting, Access, License, and Long-Term Preservation
We share data and trained model weights under Data Use Agreement (DUA) for non-commercial, research use. The Stanford AIMI Center will host and ensure long-term preservation of all data. Complete licensing terms for dataset and models are provided below.
INSPECT is available at https://stanfordaimi. azurewebsites.net/. We provide a preview subset of the entire dataset for reviewers. Before public release, the entire dataset is currently undergoing manual review by the AIMI Center to ensure no leakage of patient indentifying information.
As authors of the submitted dataset and corresponding manuscript, we hereby affirm that we take full responsibility for its contents. We ensure that this dataset and manuscript are original and that all data collection procedures were carried out ethically, respecting all relevant rights and regulations. We confirm that we have procured all necessary permissions for the use of the data included in the dataset, and that the data does not infringe upon any existing copyright, proprietary, or personal rights of others.
C.2 License Terms of Use
By registering for downloads from the INSPECT Dataset, you are agreeing to this Research Use Agreement, as well as to the Terms of Use of the Stanford University School of Medicine website as posted and updated periodically at http://www.stanford.edu/site/terms/.
Permission is granted to view and use the INSPECT Dataset without charge for personal, non-commercial research purposes only. Any commercial use, sale, or other monetization is prohibited.
Other than the rights granted herein, the Stanford University School of Medicine ("School of Medicine") retains all rights, title, and interest in the INSPECT Dataset.
You may make a verbatim copy of the INSPECT Dataset for personal, non-commercial research use as permitted in this Research Use Agreement. If another user within your organization wishes to use the INSPECT Dataset, they must register as an individual user and comply with all the terms of this Research Use Agreement.
YOU MAY NOT DISTRIBUTE, PUBLISH, OR REPRODUCE A COPY of any portion or all of the INSPECT Dataset to others without specific prior written permission from the School of Medicine.
YOU MAY NOT SHARE THE DOWNLOAD LINK to the INSPECT Dataset to others. If another user within your organization wishes to use the INSPECT Dataset, they must register as an individual user and comply with all the terms of this Research Use Agreement.
You must not modify, reverse engineer, decompile, or create derivative works from the INSPECT Dataset. You must not remove or alter any copyright or other proprietary notices in the INSPECT Dataset.
The INSPECT Dataset has not been reviewed or approved by the Food and Drug Administration, and is for non-clinical, Research Use Only. In no event shall data or images generated through the use of the INSPECT Dataset be used or relied upon in the diagnosis or provision of patient care.
THE INSPECT Dataset IS PROVIDED "AS IS," AND STANFORD UNIVERSITY AND ITS COLLABORATORS DO NOT MAKE ANY WARRANTY, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO WARRANTYES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE, NOR DO THEY ASSUME ANY LIABILITY OR RESPONSIBILITY FOR THE USE OF THIS INSPECT Dataset.
You will not make any attempt to re-identify any of the individual data subjects. Re-identification of individuals is strictly prohibited. Any re-identification of any individual data subject shall be immediately reported to the School of Medicine.
Any violation of this Research Use Agreement or other impermissible use shall be grounds for immediate termination of use of this INSPECT Dataset. In the event that the School of Medicine determines that the recipient has violated this Research Use Agreement or other impermissible use has been made, the School of Medicine may direct that the undersigned data recipient immediately return all copies of the INSPECT Dataset and retain no copies thereof even if you did not cause the violation or impermissible use.
In consideration for your agreement to the terms and conditions contained here, Stanford grants you permission to view and use the INSPECT Dataset for personal, non-commercial research. You may not otherwise copy, reproduce, retransmit, distribute, publish, commercially exploit or otherwise transfer any material.
Limitation of Use: You may use INSPECT Dataset for legal purposes only.
You agree to indemnify and hold Stanford harmless from any claims, losses or damages, including legal fees, arising out of or resulting from your use of the INSPECT Dataset or your violation or role in violation of these Terms. You agree to fully cooperate in Stanford's defense against any such claims. These Terms shall be governed by and interpreted in accordance with the laws of California.
C.3 Data Format
We detail and define our cohort in the data using a master cohort CSV file (inspect_cohort.csv), with the following primary columns.
1. patient_id: The de-identified patient id
2. procedure_time: The date of the CTPA procedure, in ISO 8601 format
3. split: A string, either "train", "valid", or "test" that indicates the data split for this patient
The primary keys for our cohort are patient_id and procedure_time. Every case, label, and feature set is associated with a patient_id / procedure_time pair.
This file also includes the following demographic columns: age, gender, race, ethnicity.
We additionally include various columns for labels.
First, we include three NLP based pulmonary embolism diagnostic label columns. pe_positive_nlp is the main PE label used in all of our experiments and described as "Positive PE" in Appendix D.1. pe_acute_nlp and pe_subsegmentalonly_nlp are the other two label columns that are similarly described in Appendix D.1. Every case in our cohort is assigned either "True" or "False" for each of these columns.
Second, we include seven prognostic label columns. These label columns correspond to the seven prognostic tasks in our paper and have the following names: 1_month_mortality, 6_month_mortality, 12_month_mortality, 1_month_readmission, 6_month_readmission, 12_month_readmission, 12_month_PH. Every case in our cohort is assigned either "True", "False", or "Censored" for each of these columns.
Finally, we include indicators for whether or not each case in this dataset is also present in either of the RNSA or Radfusion datasets using the RSNA and radfusion columns respectively.
C.3.1 CTPA
The CTPAs are made available in DICOM format. To ensure patient privacy, patient ID and study date were anonymized, and only a subset of the DICOM headers from the original DICOM file are included: ['InstanceNumber', 'ImagePositionPatient', 'PixelSpacing', 'RescaleIntercept', 'RescaleSlope', 'WindowCenter', 'WindowWidth', 'Manufacturer', 'PhotometricInterpretation', 'SliceThickness']
C.3.2 Radiologist Report Impressions
Radiologist reports with impressions sections, after the anonymization process, are included in a CSV file with the name INSPECT_anon_impression.csv.
It contains columns with the names: PatientID, StudyTime and anon_impression, where the first two columns are used to map to the master cohort file and anon_impression is the anonymized impression section.
C.3.3 Structured EHR Data
Structured EHR data is released as gzipped CSV files in the FEMR format, which is documented at https://github.com/som-shahlab/femr/blob/main/tutorials/2b_Simple_ETL.ipynb. We release all known diagnoses, procedures, lab tests, medications, visits, and death records for patients in our cohort. The FEMR format is a simplified subset of OMOP 5.3 [57], with a subset of the columns and tables. It can be processed with any CSV reader as well as with the FEMR software package.
C.4 Dataset Statistics
C.4.1 CTPA
Based on our inclusion criteria, each CTPA study can have between 50 to 600 slices. On average, each CTPA has 220.6 slices, giving us a total of 5,164,472 CT slices in our dataset. The CTPA studies range from  $1.00\mathrm{mm}$  to  $3.00\mathrm{mm}$  (Table 7) collected from CT scanners by 3 different manufacturers (Table 8).
Slice Thickness	Count
3.00	8,600
2.50	2,840
2.00	8
1.50	5,366
1.25	7,834
1.00	16,911
Table 7: Slice Thickness Distribution
C.4.2 Structured EHR Data
Distribution of history and follow-up times Our released EHR data contains all of Stanford's records for each patient in our cohort. As such, we have relatively substantial history before and follow-up time after each CTPA procedure in our cohort. Figure 5 provides the distributions for both the amount of history and the amount of follow-up time in days for our dataset. For the patients who
Manufacturer	Count
SIEMENS	11,357
GE MEDICAL SYSTEMS	3,786
TOSHIBA	3,072
Table 8: CT Scanner Manufacturer Distribution
Min	Max	Average	Standard deviation
Interval (in days)	0	6887	448.19	764.87
Table 9: Statistics of CTPA scans intervals across all patients
underwent multiple CTPA scans we also calculate some basic statistics of the intervals, shown in Table 9.
Figure 5: Patient timeline length distributions
Distribution of data types Here we show the types of data (clinical events) in the EHR patient timelines (Figure 6). We can see the measurement table dominates the OMOP table distribution and is larger than others by an order of magnitude.
C.5 Model Releases
To aid reproducibility, we release all models trained in our experiments as part of the dataset.
EHR models are in the form of pickle objects, either LightGBM Classifiers for the LightGBM models or sklearn LogisticRegression for the linear probes for MOTOR.
CT models are saved in the form of PyTorch checkpoints.
D Task Label Definitions And Validation
As part of our project, we developed and validated a set of diagnostic labels for pulmonary embolism and a set of prognostic labels for the risk of pulmonary hypertension for every case in our dataset.
D.1 Pulmonary Embolism
Task Label Definition
We construct three sets of pulmonary embolism labels ("Positive PE", "Subsegmental PE", and "Acute PE"). All the primary analysis in the benchmark is done using "Positive PE", but we release all three
Figure 6: OMOP table distribution
as part of our dataset. We define these labels using the impression section of radiology reports as ground truth. If the radiology report contains evidence of the label, we consider that to be a positive example. If the radiology report is either unclear or contains evidence against the label, we consider that a negative. We shifted the prediction time of PE to 24 hours before the CTPA exam time to avoid feature leakage, following [30].
To develop our NLP PE labeler, we use the annotated dataset described in Banerjee et al. 2019 [5]. This dataset includes 4,351 CTPA reports obtained from Stanford Healthcare Center between 2000-2016. All reports were manually annotated by board-certified radiologists according to the following labels: