SlowGuess/NeurIPS_2023_OCR huggingface dataset:
However, improvements in performance in other benchmarks, such as GLUE or BLURB, do not directly translate to improvements in RaLEs NLU. This is illustrated in Figure 3 which shows that 10 percentage point improvements in GLUE or BLURB lead to a 1-2 percent improvement in RaLEs NLU performance. We hypothesize that the decrease in benefit stems from the domain characteristics of radiology that differentiate it from other text, as proposed in Section 3. Furthermore, we observe that existing radiology domain adaptation hurts alternate domain performance, exemplified by the relatively low BLURB score for  $\mathrm{RadBERT}_{1,2}$  models seen in Figure 3. Finally, we find that given a fixed architecture, an increase in parameter count ( $O(300\mathrm{M})$  vs  $O(100\mathrm{M}))$  does not lead to improved overall performance. This observations holds for both English and biomedical models, which are the only ones publicly available in different sizes. Similar results have been previously observed in other domains [20; 45], though the impact of base model size on domain adaptation is yet to be systematically studied.
5.2 NLG
Table 4 presents the results for the report summarization task, with prior results referenced for comparison. Similarly to NLU results, no model is consistently superior across all metrics. GatorTron, the model from the clinical domain, outperforms the other evaluated fine-tuned models slightly on lexical similarity (ROUGE-2 and ROUGE-L). Using these metrics as reference, the encoder-decoder framework as implemented here performs inferiorly to the best existing performing models. The best MEDIQA 2021 prior model [10], in addition to an abstractive-summarization-specific architecture, uses a domain adaptation module to improve performance on Indiana reports. However, as can be seen from our evaluation on fully held out Stanford reports in Appendix G, most of the benefits of such specialized approaches may not generalize to sites not seen during training. Further, our approach seems to favor factual correctness, with BioLinkBERT and GatorTron models having the best performance according to the CheXbert and RG metrics. RadAdapt [49], a label efficient adaptation of a clinical large language model, matches the overall NLG performance of our best fine-tuned model. As the RadAdapt authors recognize, however, it is unclear to what extent its
Figure 2: RaLEs NLU performance by model domain. Values are averaged across model domain, error bars are standard deviation. There are no statistically significant pairwise differences between model categories within the same label availability, as determined by Mann-Whitney U-tests with Bonferroni correction.
Figure 3: General (English, left) and biomedical (right) benchmark vs RaLEs NLU performance.
performance is affected by possible leakage of testing data during the base model pre-training stage. We aim to further evaluate other large language models in future versions of RaLEs.
Table 4: Summary of results for NLG tasks (abstractive report summarization).
Model	MEDIQA 2021	BioNLP 2023	NLG score
R-2	R-L	CheXbert	RG	R-2	R-L	RG
ELECTRAbase	.238	.381	.710	.378	.156	.274	.229	.316
BioLinkBERTbase	.245	.388	.725	.391	.183	.297	.272	.337
GatorTron	.250	.386	.719	.406	.189	.303	.283	.345
RadBERT2	.237	.382	.709	.381	.184	.300	.271	.334
RadAdapt§	.253	.393	-	.345	.212	.324	.342	.345
RadiologyGPT§	.074	.148	.601	.135	.127	.209	.242	.184
Prior SOTA†	.436	.557	.718	-	-	-	-	-
Prior Baseline‡	.264	.389	.610	-	-	-	-	-
R-2/L: ROUGE 2/L, RG: F1-RadGraph,  $\dagger$ :MEDIQA[10],  $\ddagger$ :from [3],  $\S$ :our evaluation of [49] and [37]
5.3 Ethical considerations
RaLEs provides a framework for benchmarking advances in radiology NLP. The current version of RaLEs uses the RadSpRL dataset, which has a CC BY 4.0 license. Datasets stemming from MIMIC, including RadGraph, have a PhysioNetCredentialed Health Data License 1.5.0 which prohibit commercial use, data sharing and patient or institution identification attempts. For the newly released Stanza NER dataset, the dataset will be accompanied by an analogous Research Use Agreement following institutional review board approval, which was obtained to access the reports. For all datasets we have followed the appropriate research use, have not attempted re-identification of individuals, and provide instructions for data access in the accompanying code. Demographic characteristics of individuals included in RadGraph [27] and MIMIC-III [29] datasets are described in their original publications. Demographic characteristics for the Stanza NER [58], MEDIQA 2021 [3] and Indiana [16] dataset (from which the RadSpRL dataset is derived) are not reported since the radiology reports have been de-identified and these characteristics are not otherwise available. We do not foresee any potential risks following appropriate use of RaLEs as a tool to measure progress in NLP research. We strongly discourage the use of models trained using our framework for clinical care or advice without adequately studying the performance and limitations on specific patient populations that reflect intended use.
An important limitation of our analysis stems from the limited availability of publicly available radiology reports and datasets. This is consistent with prior observations that healthcare algorithms trained on US patient data rely on data from a handful of states [32]. While we release a new dataset from a new institution, future efforts should promote the availability of training and/or evaluation datasets from additional institutions, geographic locations, and languages. Aside from geographic diversity, we note that most publicly available reports and evaluations focus on chest X-ray reports which tend to be shorter and simpler than reports of other modalities and anatomies (see example lengths in Table 13). Our newly introduced procedure selection dataset expands the scope of current data by focusing on a different modality (computed tomography) across all anatomies. Finally, while we present extensive evaluations across models of different domains, we intend RaLEs to be a dynamic benchmark, with expansions across newer datasets, tasks, and model evaluations (including adding newer models and estimates of variance of performance).
Radiology reports are defined by their content, context and restricted access. RaLEs defines a benchmark for measuring progress in NLP in the radiology domain, focusing on multifaceted evaluations that reflect real-world use cases in radiology research and/or practice. The results showed that no single model, including existing radiology-specific domain adapted models, outperforms others across all evaluations.
We thank Dave Van Veen for his contribution providing results for RadAdapt summarization models. We also thank the institutions that have provided funding for this work. Research reported in this publication was made possible in part by the National Institute of Biomedical Imaging and Bioengineering (NIBIB) of the National Institutes of Health under contracts 75N92020C00008 and 75N92020C00021, GE Healthcare, and Stanford Knight Hennessy Scholars.
[1] Takuya Akiba, Shotaro Sano, Toshihiko Yanase, Takeru Ohta, and Masanori Koyama. 2019. Optuna: A next-generation hyperparameter optimization framework. In Proceedings of the 25rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining.
[2] Emily Alsentzer, John Murphy, William Boag, Wei-Hung Weng, Di Jindi, Tristan Naumann, and Matthew McDermott. 2019. Publicly available clinical BERT embeddings. In Proceedings of the 2nd Clinical Natural Language Processing Workshop, pages 72-78, Minneapolis, Minnesota, USA. Association for Computational Linguistics.
[3] Asma Ben Abacha, Yassine Mrabet, Yuhao Zhang, Chaitanya Shivade, Curtis Langlotz, and Dina Demner-Fushman. 2021. Overview of the MEDIQA 2021 shared task on summarization in the medical domain. In Proceedings of the 20th Workshop on Biomedical Language Processing, pages 74-85, Online. Association for Computational Linguistics.
[4] Aurelia Bustos, Antonio Pertusa, Jose-Maria Salinas, and Maria de la Iglesia-Vayá. 2020. Padchest: A large chest x-ray image dataset with multi-label annotated reports. Medical image analysis, 66:101797.
[5] Arlene Casey, Emma Davidson, Michael Poon, Hang Dong, Daniel Duma, Andreas Grivas, Claire Grover, Víctor Suárez-Paniagua, Richard Tobin, William Whiteley, et al. 2021. A systematic review of natural language processing applied to radiology reports. BMC medical informatics and decision making, 21(1):1-18.
[6] Pierre Chambon, Tessa S Cook, and Curtis P Langlotz. 2022. Improved fine-tuning of indomain transformer model for inferring Covid-19 presence in multi-institutional radiology reports. Journal of Digital Imaging, pages 1-14.
[7] Pierre J Chambon, Christopher Wu, Jackson M Steinkamp, Jason Adleberg, Tessa S Cook, and Curtis P Langlotz. 2022. Automated deidentification of radiology reports combining transformer and "hide in plain sight" rule-based methods. Journal of the American Medical Informatics Association.
[8] Zhihong Chen, Maya Varma, Xiang Wan, Curtis Langlotz, and Jean-Benoit Delbrouck. 2023. Toward expanding the scope of radiology report summarization to multiple anatomies and modalities. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 469-484, Toronto, Canada. Association for Computational Linguistics.
[9] Kevin Clark, Minh-Thang Luong, Quoc V. Le, and Christopher D. Manning. 2020. Electra: Pre-training text encoders as discriminators rather than generators. In International Conference on Learning Representations.
[10] Songtai Dai, Quan Wang, Yajuan Lyu, and Yong Zhu. 2021. BDKG at MEDIQA 2021: System report for the radiology report summarization task. In Proceedings of the 20th Workshop on Biomedical Language Processing, pages 103-111, Online. Association for Computational Linguistics.
[11] Surabhi Datta, Yuqi Si, Laritza Rodriguez, Sonya E Shooshan, Dina Demner-Fushman, and Kirk Roberts. 2020. Understanding spatial language in radiology: Representation framework, annotation, and spatial relation extraction from chest x-ray reports using deep learning. Journal of biomedical informatics, 108:103473.
[56] Michihiro Yasunaga, Jure Leskovec, and Percy Liang. 2022. LinkBERT: Pretraining language models with document links. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 8003-8016, Dublin, Ireland. Association for Computational Linguistics.
[57] Yuhao Zhang, Derek Merck, Emily Tsai, Christopher D Manning, and Curtis Langlotz. 2020. Optimizing the factual correctness of a summary: A study of summarizing radiology reports. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5108-5120.
[58] Yuhao Zhang, Yuhui Zhang, Peng Qi, Christopher D Manning, and Curtis P Langlotz. 2021. Biomedical and clinical english model packages for the stanza python nlp library. Journal of the American Medical Informatics Association, 28(9):1892-1899.
A MIMIC-III procedure selection dataset
The MIMIC-III dataset consists of electronic health record data of individuals admitted to critical care units at a large tertiary care hospitals [29]. Within the data made in MIMIC-III are clinical notes, among which are  $500,000+$  radiology reports. Using regular expressions on the report titles, we select 97,304 containing  $CT$  in the title. From the report header, the procedure title is extracted and procedure titles are grouped into a set of distinct procedure titles. These were manually reviewed by an MD and a board-certified radiologist, and mapped to the LOINC/RSNA Radiology Playbook $^2$ , a standardized naming and coding convention for 1,000 commonly performed radiology procedures. Each report was then labeled with an corresponding LOINC/RSNA procedure code. Procedure codes that appeared less than 100 times in the corpus were aggregated into an "Other category". This led to a total of 45 categories.
To obtain the source data for each report, regular expressions were used to extract the Clinical History section of the report. The extracted segment of text is used as a proxy for the Reason for Exam, the motivating clinical context expressed by the clinician when ordering a radiology procedure. The task consists of mapping a free-text Reason for Exam to a corresponding procedure code.
A stratified sampling strategy based on the report label was used to obtain final train/dev/test splits consisting of 60/20/20 percent of the total reports, respectively.
Figure 4: Distribution of CT procedure codes in MIMIC-III procedure selection task.
Figure 4 illustrates the distribution of report labels in the dataset. We note that the majority of entries correspond to a minority of classes, with CT Head without contrast being the most common class. Notably, there is a long tail of procedures, motivating the stratified sampling strategy. Examples of entries in the dataset include: SEIZURES  $\rightarrow$  CT Head WO contr, please eval for residual stones  $\rightarrow$
$^{2}$ https://www.rsna.org/practice-tools/data-tools-and-standards/radlex-radiology-lexicon/procedure-names-radlex-playbook
CT Abd+Pelvis WO contr, and Evaluate colon, status of ischemic colitis.  $\rightarrow$  CT Abd+Pelvis W contr IV.
In addition to stratifying by labels, our provided annotated data provides a separate dev and test sets, consisting of patients that do not appear in the train, or train  $\cup$  dev splits, respectively. The number of patients and documents in each set are detailed in Table 5. Performance of models on the set of previously unseen patients is further examined in I, where in general a decrease in performance compared to the overall test set is observed. We hypothesize that in application settings there may be a subset of patients seen during model training, as well as a subset of patients new to the institution. Therefore, it would be relevant to know how a model will perform on a mix of the two, as well as on unseen patients only as a measure of model generalizability. Thus, we provide two dev/test sets so that this difference can be kept in mind in the model design stage.
Table 5: Summary of data splits for MIMIC III Procedure dataset
Set	n Patients	n Samples
Train	19,811	58,091
Dev	11,467	19,364
Devnew only	2,170	2,603
Test	11,428	19,364
Testnew only	1,497	1,746
$\mathrm{Dev}_{\mathrm{new~only}}$  and  $\mathrm{Test}_{\mathrm{new~only}}$  are subsets of Dev and Test that only contain documents of patients not present in Train, or Train  $\cup$  Dev splits.
B BioNLP Summarization Data
Table 6: Number of reports available in the BioNLP2023 report summarization task.
Modality/Body Part	Number of Reports
Train	Dev	Test	Total
CT Abdomen-pelvis	12791	1598	798	15187
CT Chest	10228	1277	639	12144
CT Head	25121	3139	1569	29829
CT Neck	911	113	56	1080
CT Sinus	0	0	633	633
CT Spine	4413	550	275	5238
MR Abdomen	0	0	530	530
MR Head	5850	730	365	6945
MR Neck	0	0	114	114
MR Pelvis	0	0	126	126
MR Spine	0	0	1410	1410
For each DyGIE++ model fine-tuning and evaluation, we leverage the implementation by the original authors<sup>3</sup>, which is based on the allennlp and Pytorch libraries. We perform a grid hyperparameter search varying the learning rate (1e-2 - 1e-4), weight decay (0-1), and fine-tuning vs. freezing the language model layers. We use a fixed seed, 2 hidden graph layers of dimension 150, and keep other hyperparameters as described in [51].
For classification and Stanza NER, we leverage the HuggingFace transformers library. We perform hyperparameter optimization using a Tree-structured Parzen Estimator algorithm implemented by Optuna [1]. Using a fixed seed, we perform up to 10 trials varying the learning rate (1e-5 - 1e-4), number of epochs (3-5), batch size (32, 64, 128) and weight decay (1e-12 - 1e-1). We use the
<sup>3</sup>https://github.com/dwadden/dygiepp
<sup>4</sup>https://github.com/huggingface/transformers
HuggingFace evaluate library to evaluate models for F1 scores and accuracy. We use the same framework to implement the complementary metrics described in D.
For report summarization, model fine-tuning and evaluation is carried out using the ViLMedic library [13]. Models are trained using an initial learning rate of 5e-5 using the Adam optimizer [33]. A batch size of 32 with gradient accumulations over 0, 2 and 4 batches are explored. The ROUGE metric is monitored for early stopping (10 epochs). The learning rate is decayed by 0.8 if there is no improvement on the validation set for two epochs.
D Complementary metrics for NLU evaluation
It is important to evaluate models on their ability to express their uncertainty and calibration that make them suitable for deployment. Calibration ensures that estimated class probabilities match their naturally occurring prevalence. It is typically measured using the scalar summary statistic, Expected Calibration Error (ECE) [41]. Sorted predictions are divided into multiple bins (usually 10) and the absolute difference between the average accuracy and average confidence in the bin is termed the calibration gap. ECE is the weighted average of the calibration gaps [21]. Static Calibration Error (SCE) [42] was proposed as an extension to ECE to include every class in the multi-class setting, since ECE uses only the maximum probability and ignores the other class probabilities. Mathematically, the two quantities are defined as
$$
\mathrm {E C E} = \sum_ {b = 1} ^ {B} \frac {n _ {b}}{N} | \operatorname {a c c} (b) - \operatorname {c o n f} (b) |
$$
$$
\mathrm {S C E} = \frac {1}{K} \sum_ {k = 1} ^ {K} \sum_ {b = 1} ^ {B} \frac {n _ {b k}}{N} | \operatorname {a c c} (b, k) - \operatorname {c o n f} (b, k) |
$$
where  $N$  is the total number of samples,  $K$  is the total number of classes,  $n_b$  and  $n_{bk}$  are number of samples in bin  $b$  and for class  $k$ , respectively.
Weighted Model Confidence. We propose Weighted Model Confidence (WMC) as a metric that measures the quality of the model's confidence, as weighted by accuracy. This measure assigns positive weight when it's accurate (rewards confident & accurate predictions) and negative weight when inaccurate (penalizes confident & inaccurate predictions). We define it mathematically to be
$$
\mathrm {W M C} = \frac {1}{N} \sum_ {i = 1} ^ {N} \operatorname {c o n f} (i). (- 1) ^ {\operatorname {a c c} (i) + 1}
$$
where  $\operatorname{conf}(i)$  and  $\operatorname{acc}(i)$  denote confidence and accuracy of prediction  $i$ . The higher the value of this metric, the better the model is at correlating confidence with accuracy.
Average KL divergence with the Uniform Distribution (aKLU) measures the how peaked the the predicted distribution is by comparing against a flat, uniform distribution via the KL divergence, averaged over the dataset. Mathematically, it is given by
$$
\mathrm {a K L U} = \frac {1}{N} \sum_ {i = 1} ^ {N} \mathrm {K L} (\hat {p} (y _ {i} | \mathbf {x} _ {i}) | | U)
$$
where  $U$  is the uniform distribution over the classes. Higher values of the metric indicate better discriminative power of the model.
Average Predictive Entropy (aPE) measures the entropy of the predicted distribution averaged over the dataset. Lower values of aPE imply lower uncertainty in model predictions. Mathematically, it is defined as
$$
\mathrm {a P E} = \frac {1}{N} \sum_ {i = 1} ^ {N} \sum_ {k = 1} ^ {K} \hat {p} \left(y _ {i} = k \mid \mathbf {x} _ {i}\right) \log \hat {p} \left(y _ {i} = k \mid \mathbf {x} _ {i}\right)
$$
D.1 Results
We evaluate all document classification models on the MIMIC-III Procedure Selection dataset using all the complementary metrics and summarize their performance in Table 7. To provide an overview
of performance across these metrics, we rank the models based on each metric, with 1 being the highest (best) rank and 16 being the lowest. We average each model's rank across the different metrics to get the final Average Rank. We do not directly average these additional scores since, unlike F1 and accuracy scores (Table 3), these are not all percentage-based metrics. Overall, similar to what is observed with other evaluation methods, no single model outperforms all others across all tasks. General domain models have inferior performance than biomedical models, of which BioLinkBERT ranks higher across all metrics. These findings further suggest that both pretraining corpora and vocabulary, as well as improved representation learning objectives during pretraining may lead to improved performance across calibration, uncertainty and prediction quality. Notably, GatorTron, which achieves the highest accuracy, ranks 7.4/16 on average across these metrics. Alternatively, ELECTRA<sub>small</sub> achieves the best calibration, uncertainty and discriminative scores, yet has the lowest accuracy. Our proposed WMC metric, aimed at rewarding both confidence and accuracy clarifies the discordance: GatorTron achieves the best performance and ELECTRA<sub>small</sub> the worst. We believe that combining this set of metrics with Accuracy and F1 scores to the benchmark has complementary utility and is an important step towards making RaLEs model comparisons more comprehensive. We hope that these evaluations can guide future model developers to train models that are not just accurate but also well-calibrated with high prediction quality and able to express their uncertainty to enable deployment and their integration into larger systems.
Table 7: Complementary NLU Evaluations for document classification models on the MIMIC-III Procedure selection dataset using multiple families of metrics.
Model	ECE ↓	SCE ↓	aPE ↓	WMC ↑	aKLU ↑	Avg Rank ↓
BERTbase	0.065	0.004	0.994	0.339	3.897	10.8
BERTlarge	0.045	0.003	1.112	0.332	3.385	8.2
ROBERTAbase	0.040	0.002	1.144	0.320	3.472	9
ROBERTALarge	0.048	0.003	1.087	0.329	3.368	8
ELECTRASmall	0.025	0.002	1.323	0.292	3.294	7.2
ELECTRAbase	0.069	0.004	0.994	0.324	4.558	12.6
ELECTRALarge	0.046	0.003	1.121	0.321	3.722	12
DeBERTabase	0.046	0.003	1.077	0.340	3.438	6.4
DeBERTalarge	0.061	0.003	1.065	0.325	3.640	11
PubMedBERT	0.050	0.003	1.035	0.343	3.669	8
BioLinkBERTbase	0.046	0.003	1.077	0.340	3.438	6.2
BioLinkBERTlarge	0.035	0.002	1.107	0.337	3.384	5.4
BioClinicalBERT	0.047	0.003	1.089	0.337	3.377	7
GatorTron	0.061	0.003	0.977	0.355	3.556	7.4
RadBERT1	0.070	0.004	0.988	0.340	3.697	10.4
RadBERT2	0.052	0.003	1.059	0.343	3.415	6.4
Abbreviations: ECE: Expected Calibration Error, SCE: Static Calibration Error, aPE: average Predictive Entropy, WMC: Weighted model confidence, aKLU: average KL divergence with uniform distribution.
Table 8: Pearson Correlation Coefficient between added complementary metrics on NLU evaluation.
ECE	SCE	aPE	WMC	aKLU
ECE	1.00	0.93	-0.89	0.51	0.70
SCE	0.93	1.00	-0.72	0.25	0.78
aPE	-0.89	-0.72	1.00	-0.82	-0.54
WMC	0.51	0.25	-0.82	1.00	0.02
aKLU	0.70	0.78	-0.54	0.02	1.00
Abbreviations: ECE: Expected Calibration Error, SCE: Static Calibration Error, aPE: average Predictive Entropy, WMC: Weighted model confidence, aKLU: average KL divergence with uniform distribution.
E Data efficiency experiment results
Table 9: Summary of results for NLU tasks using  ${10}\%$  training and validation labels.
Model	RGNER†	RGNER‡	RGRE†	RGRE‡	RadSpRL	Stanza	Procedure	Avg
BERTbase	90.4	82.8	77.6	63.1	75.1	6.6	53.9	64.2
BERTlarge	89.7	79.7	76.2	62.6	60.1	7.0	58.8	62.0
RoBERTa-base	88.8	82.2	75.1	61.5	73.2	3.2	57.3	63.0
RoBERTaLarge	88.5	78.7	73.2	60.6	74.4	4.4	58.9	62.7
ELECTRAsmall	88.6	81.5	73.9	57.8	68.0	5.3	48.9	60.6
ELECTRAbase	92.1	79.7	75.5	58.2	71.5	8.2	54.7	62.8
ELECTRAlarge	92.5	80.5	81.2	66.3	69.5	6.5	41.6	62.6
DeBERTa-V3base	92.9	88.9	75.9	62.1	78.7	4.5	54.0	65.3
DeBERTa-V3large	90.7	81.6	75.5	63.0	79.1	7.0	49.1	63.7
PubMedBERT	91.6	82.6	78.6	64.7	77.9	5.3	60.5	65.9
BioLinkBERTbase	91.5	86.5	79.9	65.1	80.8	4.2	60.1	66.9
BioLinkBERTlarge	90.8	87.1	80.7	62.0	75.7	6.0	57.8	65.7
BioClinicalBERT	91.5	86.7	75.9	59.4	80.0	4.4	60.3	65.5
GatorTron	92.2	87.7	80.6	66.7	81.8	4.3	62.6	68.0
RadBERT1	91.1	84.5	73.1	62.8	71.3	7.2	60.9	64.4
RadBERT2	91.1	85.3	74.0	62.6	71.2	3.3	59.6	63.9
†: MIMIC-CXR (in domain), ‡: CheXpert (out of domain).