RadGraph-XL: A Large-Scale Expert-Annotated Dataset for Entity and Relation Extraction from Radiology Reports (2024)

Abstract: In order to enable extraction of structured clinical data from unstructured radiology reports, we introduce RadGraph-XL, a largescale, expert-annotated dataset for clinical entity and relation extraction. RadGraph-XL consists of 2,300 radiology reports, which are annotated with over 410,000 entities and relations by board-certified radiologists. Whereas previous approaches focus solely on chest X-rays, RadGraph-XL includes data from four anatomy-modality pairs chest CT, abdomen/pelvis CT, brain MR, and chest X-rays. Then, in order to automate structured information extraction, we use RadGraph-XL to train transformer-based models for clinical entity and relation extraction. Our evaluations include comprehensive ablation studies as well as an expert reader study that evaluates trained models on out-of-domain data. Results demonstrate that our model surpasses the performance of previous methods by up to 52% and notably outperforms GPT-4 in this domain. We release RadGraph-XL data as well as our trained model to foster further innovation and research in structured clinical information extraction 1.

heading: 1 Introduction
text: Radiology reports, which are critical for patient care, present a challenge for clinical research and applications due to their unstructured format and complex language. To address this, various methods have been developed to automatically extract important information from these reports (Langlotz
* Equal contribution 1https://github.com/Stanford-AIMI/radgraph-XL
and Meininger, 2000; Savova et al., 2010; Sugimoto et al., 2021). This is essential for tasks like training medical imaging models and monitoring diseases (Johnson et al., 2019; Irvin et al., 2019; Reis et al., 2022). However, the effectiveness of
12902
these methods is often limited by the specific types of information they are designed to extract and the scarcity of densely annotated datasets, which are costly to produce due to the need for expert input.
Recent initiatives have been directed towards addressing the complexities of deriving structured clinical data from the unstructured text of radiology reports. One such approach is RadGraph2 (Jain et al., 2021), which comprises a dataset and schema that aim to capture a wide array of clinically relevant information, such as observation and anatomical details. This schema is designed to streamline and standardize the annotation process, thereby facilitating the extraction of meaningful insights from radiology narratives. However, its application has been primarily confined to chest X-ray reports, which limits its utility across the diverse spectrum of radiology documentation (as shown in Table 1(a)). In parallel, there is a burgeoning interest within the medical AI community to extend beyond chest X-rays, exploring a wider variety of imaging modalities and anatomical regions. This expansion is evident in recent advancements in report summarization (Chen et al., 2023; Delbrouck et al., 2023), report generation (Li et al., 2022; Zhang et al., 2023a), and the development of foundation models (Wu et al., 2023b; Tu et al., 2023). These advancements underscore the necessity for innovative methodologies capable of interpreting a broader range of radiology reports.
Additionally, Large Language Models (LLMs) have also been explored to extract information with various prompting strategies (Liu et al., 2023a), beginning with a single example and expanding up to 200-shot examples to maximize the GPT-4 model’s context window. In addition to the impracticalities of the approach, including issues with access, costs, and inference time. Moreover, there is no certainty that LLMs will perform equally well across different modalities and anatomical studies, and this hypothesis remains untestable due to the lack of annotated data in these areas.
Motivated by these limitations, we introduce RadGraph-XL, a large-scale dataset featuring 2,300 radiology reports with approximately 410,000 expert annotations by radiologists (as shown in Table 1(b)(c)). These annotations cover a range of entities, relationships, and measurements across four different modality-anatomy pairs, aimed at significantly enhancing the precision and richness of
2We denote it as RadGraph-1.0 in our paper.
data extracted from radiology texts. Leveraging our annotations, we train a transformer-based model tailored for the automatic annotation of radiology reports using proven frameworks for entity and relation extraction. Our evaluation encompasses a series of ablation studies and a reader study focused on out-of-domain data, providing a thorough assessment of the model’s capabilities. Our model not only surpasses the performance benchmarks set by previous methodologies (up to 52%) but also demonstrates a significant edge over GPT-4’s capabilities in this domain (as shown in Table 1(d)).
The structure of the paper is organized as follows. First, we introduce the RadGraph-XL dataset in Section 3.1 and discuss its differences from RadGraph-1.0. Next, we outline the process of annotating the dataset and the challenges encountered in Section 3.2, and present some key statistics in 3.3. Our focus then shifts to experiments in Section 4, where we elaborate on our model’s training process (Section 4.1), our methodology for selecting the best-performing model (Section 4.2), and its comparison with a solution that employs a Large Language Model (LLM) as the transformer backbone (Section 4.3). Importantly, we highlight our model’s performance on entities defined as measurements (Section 4.4)—a novel aspect of our annotations—and compare our model’s performance with that of models from previous studies (Section 4.5). The experiments section concludes with a brief evaluation of GPT-4 against our reference test set (Section 4.6). Following this, Section 5 presents the outcomes of our reader study, and the paper concludes with Section 6.

heading: 2 Related work

heading: 2.1 Extracting Information from Radiology Reports
text: In the field of chest x-rays, traditional automated radiology report labelers, used in datasets like MIMIC-CXR (Johnson et al., 2019) and CheXpert (Irvin et al., 2019), categorize reports for common medical conditions but miss finer details like specific entities and their relationships. More detailed approaches use entity extraction schemas (Bustos et al., 2020) and focus on facts and spatial relations (Datta et al., 2020a,b), but these require dense annotation by experts. The most advanced work intended to cover most clinically relevant information within the report on chest xray is RadGraph-1.0 as discussed in Section 1. New
annotations or information extraction approaches are proposed on modalities and anatomies beyond chest x-rays, such as head CT (Jantscher et al., 2023) or chest CT (Lau et al., 2023), but remain coarse and scarce.

heading: 2.2 Downstream Tasks
text: Downstream tasks often leverage structured clinical data to enhance model performance. RadGraph1.0 (Jain et al., 2021) annotations, for example, have been utilized to boost the quality of radiology report generation by using annotations as a form of reward (Delbrouck et al., 2022), as an indicator of style (Yan et al., 2023), or to eliminate hallucinated references (Ramesh et al., 2022). They are also used in pretraining (Zhang et al., 2023b; Wu et al., 2023a), to augment the performance of fine-grained image-text self-supervised models (Varma et al., 2023), and to assess the capabilities of Large Language Models (Liu et al., 2023a; Tu et al., 2023).

heading: 3 RadGraph-XL

heading: 3.1 Overview
text: RadGraph-XL aims to enhance the capabilities of RadGraph-1.0 (Jain et al., 2021) by expanding its application across different medical imaging modalities, anatomical regions, and healthcare institutions. The proposed extensions include: • New Modality: Annotating Computed Tomog-
raphy (CT) reports for the chest, moving beyond the initial focus on Chest X-ray reports. • New Anatomy: Expanding the scope to include CT reports for the abdomen and pelvis, based on the experience with Chest CT reports.
• New Modality and Anatomy: To evaluate the model’s performance on data that is significantly different from the training set, the proposal includes annotating Brain Magnetic Resonance (MR) imaging reports, which represents a new imaging modality and anatomical region. • New Institution: Broadening the data source to include reports from a new institution from Stanford in addition to the previously used MIMICCXR reports from RadGraph-1.0. We select reports based on the following criteria in an effort to curate a diverse dataset, prioritized as follows: (i) We select reports with annotated disease labels and aim for a balanced selection to ensure an even distribution across different conditions, (ii) We employ unsupervised semantic clustering (Universal Sentence Encoder (USE) (Cer et al., 2018)) to group the reports and then select samples from each cluster, and (iii) we cluster the remaining reports by their length and sample from each cluster. A semantic projection of the USE embeddings using t-SNE is proposed in Figure 3.

heading: 3.2 Annotations
text: Each report is annotated by two board-certified radiologists. To ensure that there is a baseline level of concordance in the clinical judgments made by the two radioligists, we require the average agreement to be equal to or exceed a threshold of 50%. The average agreement rates for different imaging studies are 53.58% for Chest X-ray, 59.26% for Chest CT, 59.44% for Abdomen Pelvis CT, and 55.55% for Brain MR. If there is no consensus between the two radiologists, one judge is called upon to make a decision. In total, 406,141 annotations have been
Anatomy and Modality Type Label Chest CT Abdomen/Pelvis CT Brain MR Chest X-ray Total
Anatomy 33,976 46,326 25,104 7,715 113,121 Observation definitely present 22,425 35,595 18,033 6,469 82,522 Observation: Definitely Absent 5,705 8,975 7,215 987 22,882 Observation: Uncertain 2,104 2,867 2,121 946 8,038 Entity
validated. We use the same schema as RadGraph-1.0 to extract entities and relations from radiology reports: entities can be labeled as ‘Observation definitely present’, ‘Observation definitely absent’, ‘Observation uncertain’, ‘Anatomy definitely absent’, ‘Anatomy definitely present’ or ‘Anatomy uncertain’ and relations between entities can be labeled as ‘Located At’, ‘Modify’, ‘Suggestive Of’. For a detailed explanation of what we consider to be an entity or a relation, please refer to Appendix A. In addition to the established schema, we’ve introduced a post-processing step that identifies entities related to measurements. This effort is geared towards encouraging future research to create new models designed for dealing with or forecasting measurements, an area within radiology AI that, based on our informed understanding, presents unique challenges and has not been extensively addressed. This additional step, detailed in Section 4.4, allowed us to annotate and evaluate a subset of 3,297 entities in RadGraph-XL and 65 in RadGraph-1.0.
Finally, the task of annotating new modalityanatomy pairs presented significant challenges that are quite distinct from those encountered with chest X-rays, which were the focus of RadGraph-1.0. These complexities are detailed in Appendix B. In particular, we note that chest X-ray reports are considerably shorter than the reports for other modality-anatomy pairs, as shown in Figure 1.

heading: 3.3 Statistics
text: Table 3 provides a detailed breakdown of the annotations collected, organized by type of imaging
study and annotation categories. It is important to highlight that the dataset is evenly balanced, with anatomical annotations comprising 49.92% and observations making up 50.08%. The abdomen/pelvis CT reports, which are the longest reports in our collection as depicted in Figure 1, account for 41.38% of all annotations. This is followed by chest CTs at 28.34%, brain MRs at 23.16%, and chest x-rays at 7.11%. Regarding the types of relations anno-
Entity Relations Approach Micro F1 Macro F1 Micro F1 Macro F1
SpERT BERT 0.844±0.001 0.707±0.033 0.638±0.016 0.513±0.023 BiomedBERT 0.741±0.002 0.646±0.020 0.535±0.007 0.463±0.018 BiomedVLP-CXR-BERT 0.743±0.003 0.642±0.033 0.538±0.007 0.431±0.025
tated, 63.30% are classified as ‘modify’, 32.94% as ‘located at’, and 3.75% ‘suggestive of’. For entities, we identify 19,772 unique (entity, label) pairs; for relations, we find 67,323 (source entity, target entity, label) unique triplets. The 10 most common unique entity pairs and relation triplets are presented in Table 13.

heading: 4 Experiments

heading: 4.1 Baseline training
text: In this section, we aim to develop a predictive model using both the newly annotated dataset and the data from RadGraph-1.0. To achieve this, we evaluate two transformer-based libraries under MIT License for Entity and Relation Extraction: DyGIE++ (Wadden et al., 2019) and SpERT (Eberts and Ulges, 2020) with three proven transformer architectures, namely BERT (Kenton and Toutanova, 2019), BiomedBERT (Gu et al., 2020) and BiomedVLP-CXR-BERT (Boecking et al., 2022). We assess the effectiveness of our training through a 10-fold cross-validation process. Considering that the majority of the models discussed in this paper handle sequences of no more than 512 tokens, we ensure that both the reports and annotations are divided appropriately. The summarized results are presented in Table 4, where we demonstrate that the DYGIE++ framework delivers the best performance overall. In terms of comparing various transformer architectures, the differences observed between them are minimal. The highest-scoring transformer model
is BiomedVLP-CXR-BERT.

heading: 4.2 Selecting the Best Model
text: From the 10-fold cross-validation process, we identified the training, validation, and testing splits based on the fold where BiomedVLP-CXR-BERT achieved its highest performance, recording scores of 0.889 and 0.797 for Entity F1 Micro and F1 Macro, and 0.739 and 0.691 for Relations F1 Micro and F1 Macro, respectively. This top-performing model is now referred as to BiomedVLP-CXRBERT(b). These splits will be used as the standard for our subsequent ablation studies; for context, the selected splits for the training, validation, and test sets include 2320, 290, and 290 reports, respectively. The results of BiomedVLP-CXR-BERT(b) for entities and relationships in this specific split are detailed in the following Table 5:

heading: 4.3 Scaling with LLMs
text: The baseline architectures we selected are relatively small by current standards, each having a total of 0.11 billion parameters. In addition, we investigated transformer models with varying numbers of parameters, specifically XLM-Roberta (Conneau et al., 2019), which has 0.5 billion parameters and was trained on 2.5TB of filtered CommonCrawl data. We also looked at Pythia (Biderman et al., 2023), with 1 billion parameters trained on the Pile, and StableLM2 (StabilityAI, 2024), which has 1.6 billion parameters and was trained on a dataset of 2 trillion tokens.
Approach Entity Relations
Macro F1 Macro F1
BiomedVLP-CXR-BERT 0.796 0.689 XLM-Roberta 0.702 0.650 Pythia 1B 0.650 0.632 StableLM2 2.7B 0.789 0.656
Table 6: Comparison of BiomedVLP-CXR-BERT backbone against larger models on a 10-fold cross validation experiment.
XLM-Roberta, Pythia, and StableLM2 reports micro F1 scores that are closely matched with those of BiomedVLP-CXR-BERT, with values for entities between 0.87 and 0.88 and for relations between 0.71 and 0.73. However, they fall short in performance for certain under-represented labels, as indicated by the macro F1 scores presented in Table 6. Particularly, XLM-Roberta reports an F1 score of 0 for ‘Anatomy definitely absent’, 0.68 for ‘Observation uncertain’, and 0.47 for the relation ‘suggestive of’.

heading: 4.4 Performance on Measurements
text: The following heuristic was used to annotate measurements in the reports.
Algorithm 1 Check measurement in an entity
1 # e n t i t y i s a l i s t o f words 2 # e . g . [ " 5 " , " x " , " 5 " , "mm" ] 3 i f "mm" i n e n t i t y o r "cm" i n e n t i t y o r \ 4 "MM" i n e n t i t y o r "CM" i n e n t i t y o r \ 5 ( " x " i n e n t i t y and any (w. i s d i g i t ( ) f o r w i n e n t i t y ) ) : 6 # e n t i t y i s c o n s i d e r e d a measure
The captured measurements are highly diverse, such as ’approximately a 4.6 cm’, ’advanced by
at least 11 cm’, ’measuring slightly less than 6 mm’ or ’smaller in size compared to the prior study measuring 1.5 cm in the largest dimension’. Measurements are distributed across labels and imaging studies as follows:
Category Count
Observation definitely present 3212 Anatomy definitely present 125 Observation definitely absent 11 Observation uncertain 7 Anatomy definitely absent 4 Anatomy uncertain 3 Imaging Study Count
Stanford Abdomen/Pelvis CT 1421 Stanford Chest CT 1035 Stanford Brain MR 241 MIMIC Chest CT 225 MIMIC Abdomen/Pelvis CT 199 Stanford Chest X-ray 96 MIMIC Brain MR 80 MIMIC Chest X-ray 65
Table 7: Distribution of measurements per label and imaging study
Table 8 presents the performance metrics for these measurements as evaluated by our topperforming model, BiomedVLP-CXR-BERT(b) on the official test-set.

heading: 4.5 Comparison to RadGraph-1.0
text: To assess the value of our new annotations, we conducted two experiments.
The first experiment involves testing the model trained on RadGraph-1.0 (chest X-rays only) on our official test split, ensuring we excluded annotations labeled as ‘Anatomy Uncertain’ and ‘Anatomy definitely absent’ since they do not exist in the RadGraph-1.0 schema.
As seen in Table 9, the BiomedVLP-CXRBERT(b) model significantly outperforms RadGraph-1.0 in both categories. Specifically, in the Entity category, our model achieves a Macro F1 score of 0.863, which is approximately 16.0% higher than RadGraph-1.0’s score of 0.744. In the Relations category, the improvement is even more pronounced, with our model attaining a score of 0.691, which surpasses RadGraph-1.0’s score of 0.453 by 52.5%. These results suggest that BiomedVLP-CXR-BERT(b) provides a significantly more effective approach for recognizing entities and their relations on reports from various imaging studies. The detailed results of RadGraph-1.0 on our test-set are presented in Table 10. A significant discrepancy is observed in the category ‘Observation definitely present’, where the performance of RadGraph-1.0’s model is 20 f1-score points inferior compared to BiomedVLP-CXR-BERT(b).
In the second experiment, we trained the BiomedVLP-CXR-BERT backbone using all available data except for the official test set from RadGraph-1.0, which includes only annotations for chest X-rays. The outcomes of this experiment are detailed in Table 11. We observe that
our BiomedVLP-CXR-BERT model, despite being trained on a large, diverse dataset, can match the performance of the RadGraph-1.0 model on the RadGraph-1.0 test-set. It’s also worth mentioning that the test set is relatively small, consisting of 100 reports focused on chest X-rays. These reports are typically brief and offer limited semantic variety compared to other types of imaging studies found in our RadGraph-XL dataset.

heading: 4.6 Comparisons with GPT-4
text: Recent work has demonstrated the utility of GPT4, a task-agnostic foundation model, in effectively performing a variety of natural language tasks (Achiam et al., 2023; Liu et al., 2023b). In order to compare state-of-the-art task-agnostic models with our task-specific approach, we benchmark performance of GPT-4 on the RadGraph-XL test set. Given an input radiology report, we use GPT-4 to extract entities and relations. We evaluate performance of zero-shot GPT-4, where no in-context examples are provided, and few-shot GPT-4, where between one and ten in-context examples are included in the prompt. In-context examples are sampled randomly from the RadGraph-XL training set. Our results are summarized in Table 12.
We find that performing entity and relation extraction on the RadGraph-XL dataset is challenging for GPT-4, with macro-F1 scores observed to
be significantly lower (0.594 F1-points on entity extraction and 0.667 F1-points on relation extraction) than our task-specific approach. Few-shot GPT-4 with in-context examples exhibit slight improvements in performance over zero-shot GPT-4 (0.045 F1 points on entity extraction and 0.012 F1 points on relation extraction). In line with prior work (Liu et al., 2023a), we find that the key source of GPT-4 errors comes from incorrect understanding of the annotation schema, even in few-shot settings.
Overall, our experiments show that GPT-4 requires substantial manual prompt tuning, generates outputs that do not adequately align with the annotation schema, and requires significant postprocessing of generated outputs. Additionally, evaluations with GPT-4 are expensive, which is a particular concern on the RadGraph-XL dataset where reports are lengthy with a large number of entities and relations. Our results demonstrate (i) the need for task-specific models like our BiomedVLP-CXRBERT(b) model, which are capable of performing specialized tasks with high accuracy, and (ii) that RadGraph-XL can serve as a useful and challenging test-bed for future foundation models.

heading: 5 Reader Study
text: We conduct a reader study on out-of-domain data, namely Deep Vein Thrombosis (DVT) ultrasound reports, in order to evaluate the ability of our model to generalize to new radiological text. We chose 20 reports with semantic diversity, extracted the impressions section, and ran our top-performing model BiomedVLP-CXR-BERT(b) to predict entities and relations. Our model generated 265 entities (13.25 per report) and 207 relations (10.35 per report). A board-certified radiologist was tasked to detect critical errors, imprecise or ambiguous classifications and unclear labels, and provide a subjective overview summary.
Critical errors Three critical errors were detected. First, ‘deep’ in ‘deep veins’ was twice labeled as an observation, though it should be anatomy. This is a surprising edge case because i) our RadGraph-XL training set contains 51 ‘deep’ annotations, 40 of which are labeled as Anatomy: definitely present ii) the other ‘deep’ words were labeled correctly. Secondly, in one case, ‘some areas’ was labeled as an observation instead of anatomy (referring to some areas of the blood vessel). Finally, in one impression, ‘loss of phasicity’
and ‘loss of normal response’ were labeled as ‘definitely present’, but should have been labeled as ‘definitely absent’.
Imprecise or ambiguous classifications A few awkward labels have been predicted: ‘Thrombosis’ incorrectly modified ‘venous’ instead of indicating location. The entities ‘baker cyst’ and ‘color flow’ were wrongly marked as ‘present’ instead of ‘uncertain’, while ‘infection’ and ‘focal’ were mistakenly labeled as ‘uncertain’ rather than ‘definitely present’.
Overview summary RadGraph-XL can effectively generalize to an unknown modality and anatomic terms. For example, it was able to show that ‘spectral doppler imaging’ modifies ‘flow’, a combination of entities that is non-existent in our training dataset. Although the overwhelming majority of anatomic terms were classified correctly, there is opportunity for improvement in classifying anatomic terms, in this case “deep” as an anatomic modifier of deep vein thrombosis, that were frequently misclassified.
In summary, 5 entities out of 265 were critical errors (1.8%) and 4 entities were subjectively flagged as imprecise (1.5%). Only one relation was subjectively flagged as imprecise. Despite this study being carried on a small sample focused on ultrasound done for deep venous thrombosis, the results are encouraging for the broader use of our RadGraph-XL model for radiological information extraction.

heading: 6 Conclusion
text: We introduced RadGraph-XL, an expansive dataset comprising 2,300 radiology reports enriched with over 410,000 expert annotations from radiologists (Section 3). This dataset spans a variety of entities, relations, and measurements across multiple modality-anatomy pairs, enriching the data extracted from radiological texts with unprecedented precision and depth. We have conducted experiments (Section 4) using transformer-based models trained for automatic annotation of radiology reports, employing state-of-the-art frameworks for entity and relation extraction. Through comprehensive ablation studies (Section 4.2, 4.3, and 4.4) and a reader study (Section 5) that extends to outof-domain data, we meticulously evaluated our model’s performance. The results reveal that our model not only sets a new benchmark, but also
outperforms previous methods by as much as 52% (Section 4.5) and notably outperforms GPT-4 (Section 4.6) in this specific field. To encourage further innovation and research, we release the reports, the annotations, and our trained model.

heading: 7 Limitations
text: We denote three limitations to our work. First and foremost, the experiments have been conducted on the raw annotations without further post-processing. The annotations could be refined by implementing various heuristics to identify and address outliers in the dataset. For instance, entities with unusually long spans could be flagged for review, as these may indicate potential mislabeling or annotation errors. Similarly, entities that appear to be mislabeled could be systematically identified and corrected; Those that lack any annotations might be removed to ensure the dataset’s consistency and relevance.
Secondly, the selected transformer architectures have a maximum input size of 512 tokens, but many reports in our dataset are longer than that. It’s uncertain if dividing a report into several parts affects the model’s effectiveness due to the loss of context. Additionally, expanding the model to a size comparable to ‘Large Language Models’ and finetuning all its parameters hasn’t led to any enhancements. More advanced techniques, referred to as Parameter-Efficient Fine-Tuning (PEFT), might allow for more consistent training and the scaling up to larger models that are capable of more sophisticated reasoning.
Finally, our reader study indicates that while our model generally produces good annotations on unseen datasets, it is not immune to significant errors when dealing with out-of-distribution data. It remains uncertain how effectively our model handles unseen modalities and anatomies, and whether it can be considered reliable for annotating data in such contexts for subsequent tasks.

heading: 8 Acknowledgements
text: This work was supported in part by MIDRC (The Medical Imaging and Data Resource Center), funded by the National Institute of Biomedical Imaging and Bioengineering (NIBIB) of the National Institutes of Health under contract 75N92020D00021.

heading: 9 Ethical considerations
text: The reports subject to the annotations have been automatically deidentified with human review and approved for release by the institution. To guarantee patient safety when deploying clinical models in practice, it’s crucial for researchers training models on our datasets to rigorously audit for performance disparities across key demographic attributes, such as sex, age, and race. This involves a proactive approach to identifying and addressing potential distribution shifts that may occur when these models are applied across diverse patient populations, ensuring equitable and effective healthcare outcomes for all individuals.
We publicly release of our top-performing model, BiomedVLP-CXR-BERT(b), which is capable of automatically annotating radiology reports. Along with the model, we are also sharing our annotations with the public.

heading: B Labeling challenges
text: In the complex landscape of radiology reports, accurately identifying and annotating anatomical terms and their associated modifiers presents a significant challenge. This challenge is not only important to create high-quality labels but also crucial for maintaining consistency across reports. The nuances involved in this process can lead to variability in interpretations, which, in turn, may affect patient care and outcomes.
Anatomical Term Identification A primary concern in anatomical term identification is distinguishing between the main anatomical regions or organs and the modifiers that specify their exact locations or characteristics. An illustrative example can be seen in the description of lung scarring: ‘The lung bases are clear with the exception of some scarring in the right lung base.’ Here, ‘right’,
‘lung’, and ‘base’ are all anatomical terms. The ambiguity arises in determining whether ‘right’ modifies ‘lung’ or ‘base’, or if ‘right lung base’ should be collectively annotated as a singular anatomical entity. To mitigate such ambiguities, it is recommended that the major anatomic region or organ, in this case, ‘lung’, be labeled as the primary anatomical term. The terms ‘right’ and ‘base’ should then be annotated as modifiers that delineate the specific location within the lung.
Modifier Identification Another layer of complexity is introduced when considering how to accurately label modifiers, particularly in phrases where multiple anatomical terms are present. For instance, the phrase "There is moderate intrahepatic biliary duct dilatation" contains "intrahepatic," "biliary," and "duct" as anatomical terms. The challenge here is to ascertain whether "intrahepatic" modifies "duct" or "biliary." Consistency can be achieved by identifying the duct as the primary anatomical term and treating "intrahepatic" and "biliary" as modifiers that provide additional specificity.
Measurements A common question that arises in this context is how to handle phrases that include qualifiers such as "up to," "less than," or "greater than," which provide crucial information about the measurements being reported. Consider the sentence: "The CBD (Common Bile Duct) itself measures up to 3 cm in diameter." The use of "up to" may not be the most precise phrasing for a radiology report, where the exact measurement is typically preferred. However, the reality of clinical practice often involves approximations and ranges, particularly when exact measurements are challenging to obtain. Given their significance, it is recommended that qualifiers such as "up to," "less than," and "greater than" be labeled as observation modifiers.
Qualitative Modifiers Annotating qualitative modifiers such as ‘extensive’, ‘some’, and ‘clear’ in radiology reports presents a notable challenge. These terms significantly impact the clinical interpretation by modifying observations (e.g., ‘extensive diverticulosis’) or indicating uncertainty (e.g., ‘grossly unremarkable’). The complexity arises from their dual role in describing the severity of findings and spatial relationships between anatomical entities. Our approach recommends labeling terms that alter the interpretation of findings as observations and utilizing a generalized ‘located_at’ relation for spatial descriptors to simplify the annotation process. Terms that introduce ambiguity,
like ‘clear’ and ‘grossly’, are best represented by annotating the corresponding observations as ‘uncertain’.
Contextual Modifiers Phrases like ‘in the setting of recent surgical procedure’ or ‘hematocrit drop’ provide essential clinical context but do not directly describe imaging findings. Our guideline suggests excluding these terms from annotation, as they do not describe the radiological findings.
Compound words Determining whether to split or merge terms for annotation, such as in "hiatal hernia" or "focal pancreatitis," can be perplexing. The rule of thumb is to label words individually to maintain clarity, especially since compound terms might not always appear together in the text. However, it’s crucial to identify the primary entity in each compound term, which typically represents the main anatomy or observation. For example, "hernia" in "hiatal hernia" is the observation, with "hiatal" specifying the anatomical location. Similarly, "pancreatitis" is the observation in "focal pancreatitis," with "focal" indicating the observation’s nature.

heading: C Dataset
text: The figure below illustrates the process we used to select the reports, as detailed in Section 3.1.

heading: D Training details
text: Our best model is trained using the Entity and Relation Extraction framework DyGIE++ (Wadden et al., 2019). The parameters are defined in Table 14.

heading: E GPT-4 Evaluations
text: We provide the prompt used for GPT-4 evaluations in Figure 4.

heading: Future Work
text: Our results demonstrate (i) the need for task-specific models like our BiomedVLP-CXRBERT(b) model, which are capable of performing specialized tasks with high accuracy, and (ii) that RadGraph-XL can serve as a useful and challenging test-bed for future foundation models