SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Table 10: Summary of results for NLU tasks using  $1\%$  training and validation labels.
Model	RadGraph
RGNER†	RGNER‡	RGRE†	RGRE‡	RadSpRL	Stanza	Procedure	Avg
BERTbase	0.0	0.0	0.0	0.0	0.0	6.5	48.1	9.1
BERTlarge	2.2	1.9	0.0	0.0	0.0	4.0	40.3	7.7
RoBERTa-base	0.0	0.0	0.0	0.0	0.0	1.0	39.6	6.8
RoBERTa-large	0.0	0.0	0.0	0.0	0.0	4.4	45.5	8.3
ELECTRAsmall	1.5	1.1	0.0	0.0	0.0	6.0	35.6	7.1
ELECTRAbase	0.0	0.0	0.0	0.0	0.0	8.7	38.8	7.9
ELECTRAlarge	0.0	0.0	0.0	0.0	0.0	7.0	35.6	7.1
DeBERTa-V3base	0.0	0.0	0.0	0.0	0.9	6.1	39.1	7.7
DeBERTa-V3large	0.0	0.0	0.0	0.0	0.0	7.2	38.9	7.7
PubMedBERT	0.0	0.0	0.0	0.0	0.0	1.1	40.4	6.9
BioLinkBERTbase	0.0	0.0	0.0	0.0	0.0	4.1	35.6	6.6
BioLinkBERTlarge	0.0	0.0	0.0	0.0	0.0	6.3	37.2	7.3
BioClinicalBERT	0.0	0.0	0.0	0.0	0.0	5.8	45.2	8.5
GatorTron	0.4	0.4	0.0	0.0	0.0	5.2	55.4	10.2
RadBERT1	0.1	0.1	0.0	0.0	0.0	7.4	37.8	7.7
RadBERT2	2.4	2.0	0.0	0.0	0.0	1.3	52.0	9.2
†: MIMIC-CXR (in domain), ‡: CheXpert (out of domain).
F Linear probe experiment results
Table 11: Summary of results for NLU tasks keeping language model weights frozen during fine-tuning.
Model	RGNER†	RGNER‡	RGRE†	RGRE‡	RadSpRL	Stanza	Procedure	Avg
BERTbase	91.4	88.9	77.1	65.7	87.4	39.4	35.7	69.4
BERTlarge	91.9	88.5	72.4	57.7	82.5	13.3	35.6	63.1
RoBERTa-base	89.3	85.8	70.5	56.3	82.9	12.5	35.6	61.8
RoBERTaLarge	89.1	83.9	56.4	39.9	83.4	18.3	37.5	58.3
ELECTRAsmall	88.1	80.3	69.7	52.6	86.3	12.5	35.8	60.8
ELECTRAbase	91.1	87.1	77.0	60.1	90.0	37.6	39.0	68.8
ELECTRAlarge	90.8	85.7	69.5	56.5	88.1	33.5	38.1	66.0
DeBERTa-V3base	90.2	86.6	49.4	32.0	85.2	22.7	35.6	57.4
DeBERTa-V3large	89.9	84.8	55.4	38.8	87.3	23.0	35.6	59.3
PubMedBERT	92.9	89.4	82.1	70.5	88.5	21.9	41.8	69.6
BioLinkBERTbase	92.1	90.5	79.0	68.0	90.5	10.4	35.6	66.6
BioLinkBERTlarge	92.2	90.7	74.9	63.5	85.4	43.5	38.3	69.8
BioClinicalBERT	92.3	89.5	78.9	66.0	88.3	34.9	35.6	69.4
GatorTron	93.6	89.5	81.1	68.1	91.2	42.0	49.3	73.5
RadBERT1	91.7	87.8	76.2	61.6	88.0	16.5	51.0	67.5
RadBERT2	91.3	88.2	74.3	60.8	89.1	19.7	50.4	67.7
†: MIMIC-CXR (in domain), ‡: CheXpert (out of domain).
G Out of distribution NLG evaluation
Table 12: NLG evaluation on out of distribution (Stanford) and in distribution (Indiana) test sets.
Model	Stanford	Indiana
R-2	RG	R-2	RG
ELECTRAbase	.232	.382	.244	.374
BioLinkBERTbase	.227	.397	.263	.385
GatorTron	.238	.408	.262	.404
RadBERT2	.222	.383	.251	.379
Prior SOTA†	.277	-	.596	-
Prior Baseline‡	.241	-	.287	-
: [10], :from [3]
H Report Length Analysis
Table 13: Report length statistics for the report summarization datasets. Note that only findings and impressions sections were used to calculate report lengths.
Dataset	Split	Min	Max	Average	Standard Deviation
BioNLP2023	train	8	1279	157.5	85.3
dev	15	964	158.6	87.1
test	16	992	166.2	89.2
MEDIQA2021	train	19	430	63.5	31.0
dev	21	240	63.2	31.0
devindiana	15	230	43.1	19.7
test	18	220	57.2	31.6
I Procedure Selection Error Analysis
The following tables and figure outline error analysis for the best performing model in the benchmark, a fine-tuned GatorTron model. Evaluation metrics are obtained by studying this model across the MIMIC III Procedure selection test set.
Table 14 shows that this model may be confidently correct and incorrect. This was also quantified in Table 7, which shows metrics such as ECE. Further, example indications such as "Please evaluate for interval progression of hemorrhage" reveal that there are instances where the provided indication alone may not be sufficient to reliable determine the ground truth protocol. This observation suggests the opportunity to develop improved datasets in the future, possibly integrating prior clinical notes with the examination indications.
Table 15 shows that model performance varies by class label availability, with underrepresented classes having poorer performance in general. However, as visualized in Figure 5, this is not universally the case, as some underrepresented classes exhibit relatively higher performance with lower label availability. We hypothesize that certain classes with highly specific indications are easier to classify, even with smaller sample sizes in the training data.
Finally, Figure 6 shows the drop in performance when evaluating only new patients in the test set. The drop is observed across all model types, label availabilities and tuning strategies (full fine-tuning vs. linear probe).
Table 14: Examples of correct and incorrect predictions made by the best-performing classifier in the procedure prediction task.
Indication	Reference	Predicted	Correct?	RefP	PredP
evaluate s/p drainage of SDH	CT Head contr	WO	CT Head contr	WO	✓	0.99	0.99
evaluate for progression of subdural hematomas	CT Head contr	WO	CT Head contr	WO	✓	0.99	0.99
eval for hemorrhage post-op, 4 hours after surgery	CT Head contr	WO	CT Head contr	WO	✓	0.99	0.99
R/o IPMN	CT Abd WO+W contr IV	WO	CT Head contr	WO	×	0.00	0.99
Please evaluate for interval progression of hemorrhage.	CT Abd+Pelvis p 3D proc WO contr	WO	CT Head contr	WO	×	0.00	0.99
r/o met, bleed	CT T+L-Spine WO contr	WO	CT Head contr	WO	×	0.00	0.94
Figure 5: Per-class performance as a function of number of reports.
Figure 6: Effect of excluding patients seen during training from test set. Error bars indicate standard deviation across the models within each category.
Table 15: Per-class performance for procedure prediction task for model with highest overall accuracy. Number of samples are reported in the test set.
Procedure	Precision	Recall	F1	N
CT Head WO contr	84.3	95.3	89.4	6894
CTA Chest ves WO+W contr IV	80.8	87.3	83.9	1002
CT Guided Str loc-W contr IV	69.0	85.3	76.3	34
CT C-spine WO contr	68.5	69.5	69.0	984
CT Chest WO contr	64.3	67.3	65.7	1021
CT Abd+Pelvis W contr IV	54.5	71.1	61.7	1717
CT Sinuses+Mandible WO contr + CT Maxillofacial WO contr	67.4	51.3	58.3	298
CTA Head ves WO+W contr IV	57.9	54.2	56.0	312
CT UE WO contr	55.9	55.9	55.9	34
CT Abd+Pelvis WO contr	56.7	47.5	51.7	1037
CT Neck W contr IV	48.4	48.7	48.6	156
CTA Chest+Abd+Pelvves WO+W contr IV	43.1	50.0	46.3	170
CT Abd WO + CT Chest+Abd+Pel W contr IV	41.5	51.6	46.0	366
CTA Abd ves+Pelvis ves WO+W contr IV	38.5	54.8	45.2	155
CT Head+Brain perf+CTA Head WO+W IV	69.4	32.7	44.4	104
CT L-spine WO contr	52.7	37.1	43.6	132
CT Chest+Abd+Pelvis W contr IV	42.1	44.3	43.2	1020
CT Pelvis bones WO contr	45.7	40.0	42.7	40
CT Chest W contr IV	45.8	37.3	41.1	565
Other	49.8	33.7	40.2	978
CT Abd+Pelvis W contr IV + CTA Chest ves WO+W contr IV	55.0	31.4	40.0	210
CTA Abd Aorta+ROves-BI WO+WcontrIV	52.9	32.1	40.0	28
CT T-spine WO contr	44.4	33.9	38.5	118
CTA Head+Neck Ves WO+W contr IV	45.6	28.5	35.1	165
CT Abd WO+W contr IV	32.1	35.3	33.6	102
CTA Neck ves WO+W contr IV	36.2	29.8	32.7	57
CT Abd+Pelvis WO+W contr IV	41.0	27.0	32.6	159
CT Abd WO + CT Abd+Pel W contr IV	31.7	32.7	32.2	223
CTA Abd ves WO+W contr IV	29.3	27.5	28.4	80
CT Head WO+W contr IV	31.0	22.3	25.9	220
CT Chest+Abd+Pelvis WO contr	44.6	17.5	25.1	355
CT Abd WO contr	45.5	15.6	23.3	64
CT Head+Orbit-BI WO contr	33.3	13.6	19.4	22
CT Pelvis WO contr	26.3	11.6	16.1	43
CT Pelvis W contr IV	1	7.4	13.8	27
CTA Abd ves+Pelvis ves W contr IV	18.8	10.9	13.8	55
CT Abd+Pel + CTA AA WO+W contr IV	25.0	5.7	9.3	35
CT Neck WO contr	25.0	4.4	7.5	45
CT Chest p 3D proc WO contr	1	3.2	6.2	31
CT Guided Peritoneal Absc drain+cath plc	20.0	2.9	5.1	34
CT T+L-Spine WO contr	0	0	0	22
CT Abd W contr IV	0	0	0	45
CT Abd WO + CT Abd+Chest W contr IV	0	0	0	19
CT Abd+Pel WO + Chest+Abd+Pel W contr IV	0	0	0	32
CT Abd+Pelvis p 3D proc WO contr	0	0	0	101
CT Head W contr IV	0	0	0	53
J Report Summarization Errors
Table 16 lists example reference and generated impressions for MEDIQA2021 and BioNLP2023, the report summarization tasks examined in RaLEs. We highlight the limitations of current automated evaluation approaches. For example, when additional recommendations based on medical expertise are provided in the hypothesis, metrics such as R-L impose additional penalty on the prediction if these are not present in the reference. Conversely, in scenarios where no specific disease entities are mentioned in the ground truth, metrics like RG unfairly penalize the models given that they rely on the explicit mention of diseases and their relation to anatomies. The RG metric may be further affected by the lack of validation of RadGraph labeler on non-Chest-X-ray reports, as well as lack of normalization to a standardized nomenclature for entities. Such examples illustrate the opportunity for improved summarization evaluations. Further, they motivate importance of developing models that not only understand the immediate report but can also draw upon broader context, enabling them to offer recommendations rooted in both the findings and best clinical practice.