SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Juan M Zambrano Chaves
jMZ@stanford.edu
Nandita Bhaskhar
Maayane Attias
Jean-Benoit Delbrouck
Daniel L. Rubin
Andreas Loening
Akshay S. Chaudhari
akshaysc@stanford.edu
The radiology report is the main form of communication between radiologists and other clinicians. Prior work in natural language processing in radiology reports has shown the value of developing methods tailored for individual tasks such as identifying reports with critical results or disease detection. Meanwhile, English and biomedical natural language understanding benchmarks such as the General Language Understanding and Evaluation as well as Biomedical Language Understanding and Reasoning Benchmark have motivated the development of models that can be easily adapted to address many tasks in those domains. Here, we characterize the radiology report as a distinct domain and introduce RaLEs, the Radiology Language Evaluations, as a benchmark for natural language understanding and generation in radiology. RaLEs is comprised of six natural language understanding and generation evaluations including the extraction of anatomical and disease entities and their relations, procedure selection, and report summarization. We characterize the performance of models designed for the general, biomedical, clinical and radiology domains across these tasks. We find that advances in the general and biomedical domains do not necessarily translate to radiology, and that certain more advanced models from the general domain can perform comparably to smaller clinical-specific models. The limited performance of existing pre-trained models on RaLEs highlights the opportunity to improve domain-specific self-supervised models for natural language processing in radiology. We propose RaLEs as a benchmark to promote and track the development of such domain-specific radiology language models. RaLEs is available at https://github.com/StanfordMIMI/RaLEs.
Radiology reports convey a radiologist's interpretation of a medical image. The reports have characteristic content and structure that differentiate them from other types of text. Natural language processing of radiology reports can aid research efforts and ultimately lead to improved quality of care. However, many recent approaches focus on solving single radiology tasks, often report performance only on private datasets, fail to compare proposed methods with relevant baselines, and do not publish code or models [5]. Further, reports are typically not publicly available due to patient privacy concerns. The development of private, single-task models limits the measurement of progress in NLP for radiology broadly.
Model benchmarking in other domains, such as general English or biomedical text, has enabled thorough comparisons of existing methods across tasks that evaluate model performance and alignment with human evaluation [53; 20; 35]. This has promoted the development of state of the art models that can be adapted to address multiple tasks. Radiology reports are excluded from existing biomedical or clinical benchmarks, which often feature evaluations on datasets that do not reflect real-world clinical use cases. In contrast,  $75\%$  of current current FDA-approved AI applications target radiology ([18]), with a potential for impact on over 700 million studies (and associated reports) annually [39]. However, it is unclear how advances in other domains translate to the unique domain of radiology language and reports.
To address the aforementioned challenges, in this work we develop RaLEs, a benchmark for evaluations of natural language understanding (NLU) and generation (NLG) in the radiology domain. Our main contributions are:
1. Curating a set of 6 datasets across 4 tasks, all of which are publicly available. Among these datasets, 1 is newly created and introduced in this work (procedure selection), and 1 is newly de-identified and released into the public domain (Stanza radiology named entity recognition).
2. We benchmark and report RaLEs multi-metric performance on 16 models from the general, biomedical, clinical and radiology domains. We find that on average, clinical and radiology-specific models outperform general and biomedical models by 1.5 and  $0.5\%$  on preferred metrics.
3. We consolidate progress by developing RaLEs NLU and NLG scores, and release code for dataset standardization, model fine-tuning and evaluation to spur future benchmarking.
2.1 Radiology natural language processing
Prior work has reviewed the status of NLP in radiology. In a systematic review, Casey et al. [5] identified that  $< 20\%$  of previously published radiology NLP methods used deep learning, with the majority relying on other machine learning or rule-based systems. A significant portion of these studies primarily concentrated on tasks like information extraction (accounting for  $45\%$ ) and classification (making up  $50\%$ ). Regarding reproducibility, only 14 and 15 of the 164 studies reviewed made their data and code available, respectively. Fewer than  $20\%$  of studied compared proposed methods with alternate approaches. In a separate study, Pons et al. [44] reported that 20 of 67 studies reported operational use, the majority of which were not intended for integration into a clinical workflow.
Most existing applications of NLP in radiology can be framed as NLU or NLG tasks. These include information extraction (named entity recognition and/or relation extraction), text classification, and text summarization [40; 38]. These methods can enable applications such as extracting labels from reports to annotate images [26; 48], automatic image protocol selection, defining patient cohorts, monitoring appropriate use and clinical follow-up of medical images, and summarizing prior imaging studies [40; 38].
2.2 Benchmarks in other domains
Benchmarks that systematically compare the performance of existing models in the general English domain, such as GLUE or SuperGLUE [53; 52], have enabled comparisons of models across a variety of NLU tasks, promoting the development of pre-trained models that can be adapted to a variety of tasks using transfer learning or other adaptation methods. Other benchmarks have been proposed for biomedical language understanding and reasoning, such as the BLURB Benchmark [20], which contains 13 datasets and 6 tasks, focusing on the performance of systems across scientific biomedical text. The Biomedical Language Understanding Evaluation (BLUE) benchmark [43], includes 6 biomedical scientific text datasets and 4 clinical datasets evaluating sentence similarity, named entity recognition, relation extraction, document classification and inference. While comprehensive, the BLUE benchmark does not include any radiology text-based tasks. This highlights a broader issue: no existing benchmarks broadly compare the performance of models in a curated set of radiology tasks, which makes it challenging to quantify the efficacy of general-purpose language models on domain-specific radiology tasks.
3 Radiology reports as a domain
Three key characteristics define the domain of radiology reports: a) content, b) context and c) closed source. In terms of content, radiology reports contain a limited specialized vocabulary, often existing only in the context of images. For example, words like pneumothorax, cardiomegaly, and radiodensity, referring to air in the space surrounding the lungs, an enlarged heart, and opacity to X-rays, are frequently found in radiology reports. Such words are contained in Radiology Lexicon<sup>1</sup>, a comprehensive set of radiology terms that contains approximately 30,000 entries. An additional aspect of content involves structure. Radiology reports typically follow a document structure comprised of a header containing patient history and exam-related information, followed by mentions of relevant comparison studies, details about the imaging technique utilized, a detailed description of image findings, and an impression that summarizes findings contextualized to the patient's condition [30; 23]. Sentences within reports are typically short, declarative and factual, and are written in the present tense. The content (unique vocabulary, format, and narrative style) of radiology reports is an important feature of this specialized domain.
Another aspect that characterizes this domain is the context surrounding radiology reports. Reports only exist in the presence of an accompanying medical image. Typically they exist within the context of electronic medical records, which are collections of documents, images and other signals that describe a patient's medical history. The content of these reports is embedded within the current radiology and medical knowledge.
Finally, due to existing concerns and regulations that protect patient health information, the vast majority of radiology reports exist within private data warehouses, often requiring institutional review board approval for access. The largest publicly available collections of reports, such as MIMIC-III, MIMIC-CXR, PadChest, and Open-i datasets [29; 28; 4; 15], have undergone de-identification and are typically made available subject to agreement to terms of data use protecting patient privacy.
4 RaLEs: Radiology Language Evaluations
The following sections outline RaLEs. The tasks and datasets were selected to reflect real-world use cases, ensuring they are both challenging and achievable. We outline multiple metrics of success for well-rounded evaluation for each dataset-task pair, as well as differentiate the performance of the model in data from institutions unseen during training where available.
Table 1: Overview of datasets and tasks in RaLEs.
Category	Dataset	Task	# Train/Dev/Test	Anatomy/Modality	# Sources	New
NLU	RadGraph [27]	RE	425 / 75 / 100	Chest / XR	2
RadGraph [27]	NER	425 / 75 / 100	Chest / XR	2
RadSpRL [11]	RE	848 / 105 / 106	Chest / XR	1
Stanza Radiology [58]	NER	2,461 / 200 / 295	Chest / CT	3	✓
CT Procedure Selection	Clf.	58,091 / 19,364 / 19,364	Varied / CT	1	✓
NLG	MEDIQA 2021[3]	Summ.	91,544 / 4,000 / 600	Chest / XR	2
BioNLP 2023[14]	Summ.	59,320 / 7,413 / 13,057	Varied / CT, MR	1
XR=X-ray, CT=Computed Tomography, MR=Magnetic Resonance Imaging, NLU=Natural Language Understanding, NLG=Natural Language Generation, New= newly released with this work, Varied=Abdomen, Pelvis, Neck, Spine, Head, etc., see A and B
Table 1 summarizes tasks and datasets within RaLEs. We include four tasks as part of the initial RaLEs: named entity recognition (NER), relation extraction (RE), document classification and document summarization. The datasets selected for each task are described as follows.
RadGraph [27] consists of 600 manually annotated chest x-ray reports from MIMIC-CXR and CheXpert datasets [28; 26]. A board-certified radiologist labeled the reports with named entities, consisting of Observation - definitely present, Observation - definitely absent, Observation - uncertain, and Anatomy, reflecting key entities in reports corresponding to observations (which may be negated or hedged) as well as anatomical structures. Pairs of entities may be related by one of three types: suggestive of, located at, and modify. We use this dataset to evaluate models on NER and RE. Test
<sup>1</sup>https://www.radlex.org
set evaluations are carried out separately on MIMIC-CXR and CheXpert, reflecting in and out of distribution performance, respectively.
RadSpRL [11] consists of 2000 manually annotated chest x-ray reports from the Open-i dataset [15]. A medical librarian and an MD annotated entities and relations that represent spatial relations. Spans of text were annotated as a relationship between a Spatial indicator with another span of text consisting of one of four spatial roles: Trajectory, Landmark, Hedge, and Diagnosis. We evaluate models on RE with this dataset. We use only documents with labeled relations for training and evaluation. Though prior performance is reported on cross-validation sets, we create a fixed train/dev/test split on the report level to limit excessive compute requirements of hyperparameter exploration across each split for each model.
Stanza Radiology [58] consists of 150 manually annotated chest computed tomography reports from three hospitals. Two radiologists annotated spans of text in the reports with five entity types: Anatomy, Observation, Anatomy modifier, Observation modifier, and Uncertainty. This dataset is used for NER evaluations. As part of RaLEs, we have deidentified these reports using a publicly available hidden-in-plain-sight de-identification algorithm [7], and release this previously private dataset to the public.
MIMIC-III procedure selection is a newly created dataset, released alongside this work, that consists of 96,819 documents extracted from individual computed tomography reports from the MIMIC-III dataset [29]. The reason for exam and procedure title were extracted from each report using regular expressions. The task consists of appropriately classifying reason for exam documents into one of 46 normalized procedure titles. The procedure titles were normalized to a standardized vocabulary [50] by manually mapping a set of extracted procedure titles to the vocabulary. Normalization was carried out by an MD and a board-certified radiologist. Additional details of the curation of this dataset are in A. This evaluation was developed to simulate the selection of a procedure given a clinician provided reason for exam, a task that often requires expert human oversight in current practice.
MEDIQA 2021 report summarization [3] consists of 96,144 chest X-ray reports with extracted Findings and Impression sections. The task is summarizing the Findings section of reports, using the Impression as ground truth. Models are trained using reports from one dataset (MIMIC-CXR) and validated using reports from MIMIC-CXR and the Open-I dataset (from Indiana). The test evaluation is carried out on reports from an institution seen during validation (Indiana), as well as an institution present only in the test set (Stanford), which aims to measure out-of-domain generalization.
BioNLP 2023 report summarization [14] consists of 79,790 multi-modal reports [8] extracted from the MIMIC-III dataset [29] that are separated into Findings and Impression sections. The task is to create a summary in the same fashion as MEDIQA 2021. The reports are of computed tomography and magnetic resonance imaging examinations, with head, chest, abdomen, spine and sinuses present as different anatomies (Details in B. Test evaluations include anatomies/modalities unseen in training data.
4.2 Evaluation strategy
4.2.1 Models
Table 2: Masked language models evaluated. Number of parameters in millions.
Domain	Model	# Params
English	BERT	{110, 340}
RoBERTa	{125, 355}
ELECTRA	{14,110,345}
DeBERTa-v3	{86, 304}
Biomedical	PubMedBERT	110
BioLinkBERT	{110, 340}
Clinical	BioClinicalBERT	110
GatorTron	345
Radiology	RadBERT1	110
RadBERT2	125
We evaluate various pre-trained masked language models from the general, biomedical, clinical and radiology domains. Table 2 lists the models evaluated. We refer the reader to the respective publication for details on the pretraining strategy including source of vocabulary, corpus and model-specific optimizations.
From the general English domain, where models are trained using sources such as Wikipedia and Google Books, we evaluate BERT [17] in its base and large configurations, RoBERTa [36] in its base and large configurations, ELECTRA [9] in its small, base and large configurations, and DeBERTa-v3 [24] in its base and large configuration.
From the biomedical domain, we examine models pre-trained on scientific text contained in PubMed: PubMedBERT-base [19], which pretrains a BERT-base model using biomedical scientific abstracts and full-text articles as a corpus, and BioLinkBERT-base and large [56], which uses a similar pretraining corpus but incorporates an additional pretraining objective consisting of identifying document links.
From the clinical domain, we evaluate BioClinicalBERT [2], a BioBERT [34] biomedical model (i.e., a BERT-base model continually pretrained on PubMed text), which is further continually pretrained on MIMIC-III clinical notes. In addition, we examine GatorTron-base[55], a MegaTronBERT model [47] pretrained using clinical notes from the University of Florida, PubMed text and Wikipedia articles (500GB of text).
Finally, we examine the performance of models developed specifically for the radiology domain, both named RadBERT by their creators. One model, which we refer to as  $\mathrm{RadBERT}_1$ , corresponds to a BioBERT model continually pretrained on 4 million radiology reports from Stanford Health Care [6]. The second model, which we refer to as  $\mathrm{RadBERT}_2$ , corresponds to a BioMed-RoBERTa (RoBERTA-base model continually pretrained on 2.7 million scientific papers [22]) model further pretrained on 4.4 million radiology reports from various facilities of the U.S. Department of Veterans Affairs health system.
4.2.2 NLU evaluations
We fine-tune each model using all documents in training split and perform task-specific hyperparameter optimization as detailed in C. We select the best model according to the best run preferred metric on the validation set, as defined by each dataset. We report the performance of models on test sets using the best model for each model type/task. We employ different fine-tuning strategies for each task, described as follows.
For RE, datasets (RadGraph, RadSpRL), we use DyGIE++ [51], a multi-task NER and RE framework that learns a dynamic graph that models relationships between text spans. Span representations are obtained from a language model embedding. We use DyGIE++ as a NER extraction method for RadGraph models as we empirically observed improved performance compared to the Stanza NER approach which cannot leverage the relation labels. For document classification, a randomly initialized head of dimensions  $(h,c)$  where  $h$  corresponds to the hidden size and  $c$  corresponds to the number of classes, is added as a final layer to classify the classification [CLS] token. For Stanza NER, a randomly initialized head of dimensions  $(h,c)$  is added to classify each token. For words comprised of multiple sub-word tokens, the label is assigned to the first token.
Consistent with prior evaluations for existing datasets, the preferred metric for all evaluations is the micro-averaged F1 score. For the newly created procedure selection task, we select accuracy as the preferred metric as it more closely reflects success in clinical settings. We report a RaLEs NLU score as the average of the preferred metrics across the NLU datasets. In addition, we perform label-stratified sampling evaluations using  $1\%$  and  $10\%$  labels during training and validation to assess how dataset scaling across domain-specific and domain-agnostic models affects their performance. We also examine separability of representations generated by each model by keeping language model weights frozen and training only a linear probe, or only the graph layers in the case of DyGIE++ models. Furthermore, we evaluate models in metrics that may be relevant for deployment, focusing on model calibration and uncertainty. We include in RaLEs metrics measuring calibration and uncertainty, prediction quality, and information criteria. We detail the results for the document classification (procedure selection) task on these metrics in Appendix D. Model fine-tuning is performed in private infrastructure using a single NVIDIA Titan RTX or RTX A6000 GPU. For each model, hyperparameter exploration takes on average 1 hour for DyGIE++ models, 1 hour for Stanza NER, and 3 hours for the procedure selection task (in total approximately 400 GPU hours).
4.2.3 NLG evaluations
For report summarization, we fine-tune each model to predict the Impressions section of a radiology report given the Findings, using the MEDIQA2021 and BioNLP 2023 datasets. Specifically, we construct an encoder-decoder model for sequence generation [46] using the same pretrained model as the encoder and the decoder, and newly initialize a cross-attention layer. All weights are updated during model fine-tuning. Due to increased computational cost of hyperparameter exploration for each model (10-20 hours per training run on an RTX A6000 GPU), we chose a representative model from each domain for study: ELECTRA<sub>base</sub>, BioLinkBERT<sub>base</sub>, GatorTron, and RadBERT<sub>2</sub>. Further, we include our evaluations of two recently proposed report summarization-specific approaches: RadiologyGPT and RadAdapt [37; 49].
We evaluate the quality of generated summaries using commonly used lexical similarity metrics, ROUGE-2, ROUGE-L [31; 25], as well as recently proposed metrics to evaluate factual correctness of a report [57; 12]. For factual correctness in Chest X-ray report summarization (MEDIQA 2021), we examine the F1-CheXbert score, which measures the micro-averaged F1 score of 14 disease mentions extracted from a generated summary, using the original as ground truth [57]. In addition, we report the F1-RadGraph [12] (RG) metric, which compares the agreement of anatomy and observation entities and their relations extracted in the generated versus the original summary. We report the test-set results using the ROUGE, CheXbert and RG metrics, and summarize model performance in the RaLEs NLG score, which is the average of the ROUGE-L and RG metrics across both datasets.
5.1 NLU
Table 3: Summary of results for NLU tasks.
Model	RGNER†	RGNER‡	RGRE†	RGRE‡	RadSpRL	Stanza	Procedure	NLU Score
BERTbase	93.0	86.3	82.7	70.6	91.2	83.7	65.4	81.8
BERTlarge	92.6	88.5	82.0	71.4	85.3	85.4	64.9	81.4
RoBERTa_base	92.4	89.7	81.3	70.2	89.6	81.5	64.2	81.3
RoBERTaLarge	92.7	89.7	83.0	73.1	82.1	83.0	64.9	81.2
ELECTRAsmall	92.6	89.7	82.0	70.7	88.1	73.1	61.5	79.7
ELECTRAbase	93.2	85.9	83.2	71.9	89.9	85.2	64.4	82.0
ELECTRAlarge	93.0	86.0	82.3	71.0	87.5	84.9	64.2	81.3
DeBERTa-V3base	93.4	89.8	84.6	73.4	89.8	84.1	65.7	83.0
DeBERTa-V3large	93.1	89.9	83.8	73.4	89.4	85.1	64.2	82.7
PubMedBERT	92.1	86.8	82.9	71.7	88.9	85.0	65.9	81.9
BioLinkBERTbase	93.2	90.6	83.6	75.1	91.0	83.7	65.7	83.3
BioLinkBERTlarge	93.2	90.3	82.6	72.4	89.6	84.6	65.7	82.6
BioClinicalBERT	93.7	90.4	82.0	72.8	91.9	85.6	65.4	83.1
GatorTron	93.5	89.8	82.9	74.6	92.0	84.3	66.7	83.4
RadBERT1	93.8	90.1	81.2	72.1	91.2	85.1	65.4	82.7
RadBERT2	94.0	90.7	81.9	73.4	91.2	85.5	65.8	83.2
Prior SOTA§	94.0	90.5	82.3	72.5	85.6	84.8	-	-
†: MIMIC-CXR (in domain), ‡: CheXpert (out of domain), §: RadGraph [27], RadSpRL [11], Stanza [58]
Table 3 summarizes the results of the NLU evaluation. No single model outperforms all others across all tasks. In general, biomedical or clinical models outperform general domain models. Furthermore, most models perform similarly (within 2-3 percentage points of each other; standard deviation of average performance across models is 1).
5.1.1 Impact of domain, pretraining objective and model size
Figure 1 shows the summarized performance of models across domain and size. Overall, general English models perform worse than their more specialized counterparts. Furthermore, models in the clinical/radiology domain outperform those from the biomedical and general domains. This is consistent with prior observations of benefits of using domain specific corpora during pretraining[19; 55; 54]. This trend is consistent across varying availability of labeled fine-tuning data (Figure 2).
Additionally, improved pretraining objectives typically lead to improvements in RaLEs NLU performance. For example, ELECTRA models (adversarily trained to predict replaced tokens) outperform models trained only with masked language modeling objectives (such as BERT). DeBERTa-V3, which in addition to replaced token detection enhances token representation with disentangled attention,
Figure 1: Model size and domain vs RaLEs NLU performance
obtains improvements in both in GLUE as well as in RaLEs NLU. Similarly, though BioLinkBERT and PubMedBERT share similar scientific pretraining corpora, the document relation prediction objective of BioLinkBERT leads to improved performance in both BLURB as well as RaLEs NLU.