Extracting and Encoding: Leveraging Large Language Models and Medical Knowledge to Enhance Radiological Text Representation (2024)

Abstract: Advancing representation learning in specialized fields like medicine remains challenging due to the scarcity of expert annotations for text and images. To tackle this issue, we present a novel two-stage framework designed to extract high-quality factual statements from freetext radiology reports in order to improve the representations of text encoders and, consequently, their performance on various downstream tasks. In the first stage, we propose a Fact Extractor that leverages large language models (LLMs) to identify factual statements from well-curated domain-specific datasets. In the second stage, we introduce a Fact Encoder (CXRFE) based on a BERT model fine-tuned with objective functions designed to improve its representations using the extracted factual data. Our framework also includes a new embedding-based metric (CXRFEScore) for evaluating chest X-ray text generation systems, leveraging both stages of our approach. Extensive evaluations show that our fact extractor and encoder outperform current state-of-the-art methods in tasks such as sentence ranking, natural language inference, and label extraction from radiology reports. Additionally, our metric proves to be more robust and effective than existing metrics commonly used in the radiology report generation literature. The code of this project is available at https://github. com/PabloMessina/CXR-Fact-Encoder.

text: Findings of the Association for Computational Linguistics ACL 2024, pages 3955–3986 August 11-16, 2024 ©2024 Association for Computational Linguistics
Advancing representation learning in specialized fields like medicine remains challenging due to the scarcity of expert annotations for text and images. To tackle this issue, we present a novel two-stage framework designed to extract high-quality factual statements from freetext radiology reports in order to improve the representations of text encoders and, consequently, their performance on various downstream tasks. In the first stage, we propose a Fact Extractor that leverages large language models (LLMs) to identify factual statements from well-curated domain-specific datasets. In the second stage, we introduce a Fact Encoder (CXRFE) based on a BERT model fine-tuned with objective functions designed to improve its representations using the extracted factual data. Our framework also includes a new embedding-based metric (CXRFEScore) for evaluating chest X-ray text generation systems, leveraging both stages of our approach. Extensive evaluations show that our fact extractor and encoder outperform current state-of-the-art methods in tasks such as sentence ranking, natural language inference, and label extraction from radiology reports. Additionally, our metric proves to be more robust and effective than existing metrics commonly used in the radiology report generation literature. The code of this project is available at https://github. com/PabloMessina/CXR-Fact-Encoder.

heading: 1 Introduction
text: In the context of medical image analysis, radiology reports serve as a rich source of information. Radiologists routinely generate these free-text reports, which typically include sections such as comparison, indication, findings, and impression, as illustrated in Figure 1.
Radiology reports have been employed for various purposes, including label extraction for structured supervision in medical image tasks (Irvin et al., 2019; Wu et al., 2021; Jain et al., 2021a),
training data for models in report generation (Messina et al., 2022; Miura et al., 2021; Tanida et al., 2023), summarization tasks (Chen et al., 2023b; Ma et al., 2023), and the development of multimodal models capable of jointly understanding medical images and text (Wang et al., 2022; Boecking et al., 2022; Bannur et al., 2023).
A crucial aspect in addressing such tasks is the accurate comprehension of factual information within the report. Specifically, the findings and impression sections can be considered as repositories of factual information regarding the imaging examination. These factual statements encompass various elements, including observations (such as abnormalities, diseases, or devices), interpretations derived from observations, references to anatomical locations, discussions on severity or confidence levels, comparisons to previous studies, and more. For instance, in Figure 1, one factual statement indicates no acute bone abnormality (a normal observation), while another describes a stable calcified granuloma within the right upper lung (an abnormality found in a specific anatomical site).
Despite the various aforementioned applications that recent research has explored in the use of radiology reports, the persistent absence of effective methods for precise fact extraction and encoding from medical reports remains a critical challenge. As demonstrated in our experimental evaluation (Section 4), existing encoders and label extraction techniques frequently struggle to capture the nuanced details within free-text radiology reports, re-
3955
sulting in incomplete or inaccurate depictions of clinical information. This is evident in various aspects. For instance, existing text encoders developed for the medical domain may struggle to generate consistent representations of paraphrased statements (Table 1) or to differentiate between similar sentences conveying contradictory meanings (Table 2), a crucial requirement to prevent encoding erroneous diagnoses. Similarly, current label extraction methods often rely on rigid labeling schemes based on manually crafted rules, leading to incomplete capture of all factual statements within a report (Table 5). Similar limitations are also observed in commonly used evaluation metrics for radiology text generation (Table 4).
In this work, we propose a novel approach that leverages the capabilities exhibited by Large Language Models (LLMs) such as ChatGPT, which have showcased outstanding performance in medical contexts (Liu et al., 2023c; Katz et al., 2023; Liu et al., 2023b; Adams et al., 2023), to improve factual statement representation. Our methodology also takes advantage of the existing knowledge in expert-annotated datasets. These datasets offer indispensable training data and also serve as a benchmark to enhance our model’s clinical terminology and context comprehension. Concretely, our contributions are three-fold:
• A fact extractor: a novel and simple approach to extracting facts that leverages ChatGPT and a fine-tuned version of T5 (Raffel et al., 2020) to capture relevant information from Chest X-ray radiology reports without requiring annotations from radiologists.
• A fact encoder: Chest X-ray Fact Encoder (CXRFE), a CXR BERT-based model (Boecking et al., 2022) fine-tuned with a multitask approach that leverages domain expertise from radiologists as well as ChatGPT and T5 generated annotations. CXRFE exhibits significant improvement in fact comprehension, demonstrated on sentence ranking and natural language inference tasks.
• A new evaluation metric for radiology text generation: CXRFEScore, which measures the factual accuracy of a generated text relative to a real text, by extracting and comparing the similarity of fact embeddings.
We release the weights of all our models, as well as the data and code necessary to replicate the
results. We also release CXRFEScore as a Python library for ease of use by the research community.

heading: 2 Related Work
text: In this section, we discuss prior work on BERTbased approaches to radiology text representation and label extraction from radiology reports, and leave discussion of prior work on evaluation of factual correctness in radiology text generation, applications of LLMs to medical text, and knowledge distillation from LLMs to Appendix A.1.
BERT-based Approaches for Radiology Text Representation. The advent of BERT (Devlin et al., 2019) has sparked notable progress in numerous NLP domains. This has inspired researchers to customize BERT for specific applications, including the medical field. Pioneering works such as BioClinicalBERT (Alsentzer et al., 2019), PubMedBERT (Gu et al., 2020), and BioLinkBERT (Yasunaga et al., 2022) have applied the masked language modeling (MLM) objective introduced by BERT to domain-specific corpora, such as PubMed paper abstracts and MIMIC-III (Johnson et al., 2016), an electronic health records dataset.
More recently, specialized variants like CXRBERT (Boecking et al., 2022) and BioViL-T (Bannur et al., 2023) have been developed, targeting the unique challenges posed by CXR reports. CXRBERT provides both a general version, pretrained with MLM on PubMed abstracts and MIMICIII documents, and a specialized version, finetuned with MLM coupled with a radiology section matching loss specifically tailored for reports from the MIMIC-CXR dataset (Johnson et al., 2019b). BioViL-T adopts the same pretraining strategy as CXR-BERT but is subsequently fine-tuned using global and local multi-modal contrastive learning and image-informed MLM objectives. By combining reports with temporally sequenced image pairs, this approach enhances the understanding of radiological sentences with temporal descriptions.
Drawing inspiration from these works, we adopt BERT as our base model for text encoding. However, unlike prior approaches that aim to improve BERT’s representations with a single pre-text task (Reimers and Gurevych, 2019; Araujo et al., 2023), we employ a novel domain-specific multi-task learning protocol. This protocol leverages LLMs to generate large-scale supervision alongside expertcurated annotations from domain-specific datasets.
Label Extraction from Radiology Reports. Our work is also related to the problem of extracting information, usually in the form of labels, from free-text radiology reports. A well-known example in the literature is the CheXpert labeler (Irvin et al., 2019), which uses a rule-based system to infer the presence or absence of 13 observations (plus the label "No findings"). CheXbert (Smit et al., 2020) and VisualCheXbert (Jain et al., 2021b) are subsequent versions that follow the same labeling standard of CheXpert but are based on BERT.
The Chest ImaGenome (Wu et al., 2021) dataset is another example that used a rule-based NLP system to label reports to construct scene graphs for the corresponding frontal images in the MIMICCXR dataset (Johnson et al., 2019a). RadGraph (Jain et al., 2021a) proposed a labeling scheme of entities and relations for radiology reports and trained a variant of BERT, DyGIE++ (Wadden et al., 2019), for entity and relation extraction on examples annotated by radiologists. PadChest (Bustos et al., 2019) followed a similar approach, by labeling reports with an LSTM that was previously trained on examples annotated by physicians.
Our work contributes to this field by introducing a more flexible, open-vocabulary approach to information extraction, focused on extracting the essential factual information contained in the report, without imposing constraints that are too rigid. Specifically, we propose extracting factual statements, referred to as "facts," from reports, by leveraging the proven effectiveness of recent LLMs.

heading: 3 Method
text: We introduce a two-stage method for encoding the information within a CXR report. In the first stage, called fact extraction (Section 3.1), we utilize LLMs to extract facts from the original sentences of the report. In the second stage, called fact encoding (Section 3.2), we employ a BERT-based text encoder to generate sentence embeddings for each extracted fact. When used in tandem, these two stages form a cohesive system capable of producing vectorial representations of the factual statements found within a CXR report.

heading: 3.1 Fact Extraction
text: Figure 2 outlines our method for extracting facts from radiology reports, with examples taken from the MIMIC-CXR dataset (Johnson et al., 2019b). Initially, we use regular expressions and simple
rules to pinpoint relevant radiological sections in MIMIC-CXR reports, mainly Findings and Impression, but we also handle alternate headings. These sections are then divided into sentences. For simplicity, we use the sent_tokenize function from the NLTK library1. Next, we proceed to extract concise factual statements from each sentence. The main reason for doing this is that radiologists often write sentences that are noisy or complicated. Figure 2 shows two examples of such sentences. The first example contains multiple factual statements in one sentence, which can be simplified into shorter phrases. The second example is overly verbose, but the essential observation can be summarized in a brief phrase. We provide more examples in Table 6. Given the recent success of LLMs, an effective strategy to achieve this sort of extraction is by directing ChatGPT using a custom prompt.
T5 as an alternative to ChatGPT. In theory, this entire process could be executed using off-the-shelf LLMs. However, the expenses associated with accessing the API to annotate the entire dataset can be prohibitive. Therefore, the alternative approach we adopted was to annotate only a strategically selected subset of sentences and then transfer the acquired knowledge from these annotations to a more cost-effective sequence-to-sequence model, such as T5, through fine-tuning. This approach mirrors the strategy employed by Yang et al. (2023), where a T5 is fine-tuned to condense verbose descriptions from GPT-3 in LLM-assisted image classification.
We provide detailed implementation steps for this fact extraction procedure in Appendix A.2

heading: 3.2 Fact Encoding
text: After we extract facts, we encode them to obtain vectors in a latent space via a text encoder model, called CXRFE. In this work, we rely on CXRBERT (Boecking et al., 2022) to implement our fact encoder. Specifically, we use the CXR-BERTspecialized variant available on the Huggingface hub2, leveraging its built-in [CLS] token projection, which yields a 128-D vector serving as the final representation of the text.
Building on top of CXR-BERT-specialized, we explore 6 different training tasks to enhance the latent representation of radiological sentences: triplet loss for sentence ranking (T), natural language in-
1https://www.nltk.org/ 2https://huggingface.co/microsoft/
BiomedVLP-CXR-BERT-specialized
Figure 3
ference (NLI), quadruplet loss to enforce a separation between entailment and contradiction pairs (EC), entity and relation extraction (ER), sentence classification tasks (C), and sentence decoding (SD). We provide details on the implementation of each task in Appendix A.3.
Thus, by combining the two stages, the whole framework can accurately extract and encode facts from CXR reports, thus providing a rich and convenient representation of the factual information for downstream applications.

heading: 3.3 CXRFEScore
text: A potential application of our framework is in evaluating report generation from chest X-rays. We introduce CXRFEScore, an embedding-based metric that leverages both T5 and our CXRFE model for evaluation. Figure 3 illustrates how the metric works.
Given a reference report and a candidate report, we extract facts from each and represent them as embedding vectors, denoting the sets for the reference and candidate reports as R and C respectively. The cosine similarity matrix M of size |R| × |C| is formed, where Mi,j represents the cosine similarity between the ith vector of R and the jth vector of C. This allows us to calculate Srow, Scol, and the final CXRFEScore as follows:
Srow = ∑ i maxj Mi,j |R| Scol = ∑ j maxi Mi,j |C|
CXRFEScore = Srow+Scol2 The equations of this metric resemble those of BERTScore (Zhang et al., 2020a). The key difference lies in CXRFEScore’s comparison of fact embeddings rather than token embeddings. This metric illustrates the fusion of the two proposed
stages: fact extraction (implemented with T5) and fact encoding (implemented with CXRFE). We provide evidence of the robustness of this metric compared to many existing metrics in Section 4.

heading: 3.4 Dataset construction
text: In our experiments, we primarily utilize the MIMIC-CXR dataset (Johnson et al., 2019b), which comprises 227,827 radiology reports associated with 377,110 chest X-ray images. However, we focus solely on utilizing the reports for our experiments, deferring the exploration of images and multi-modality for future work. Additionally, we incorporate annotations from the Chest ImaGenome dataset (Wu et al., 2021), which provides scene graphs linking observations to anatomical image locations for each frontal view image in MIMIC-CXR. These annotations serve two main purposes: facilitating our creation of a binary multi-label classification task and introducing a radiologist-informed annotation standard covering various observation types and anatomical locations. Similarly, we utilize RadGraph (Jain et al., 2021a), which offers an entity and relation annotation scheme for radiology reports in MIMIC-CXR, and datasets such as MedNLI (Romanov and Shivade, 2018), RadNLI (Miura et al., 2021) and MSCXR-T (Bannur et al., 2023) for experiments on Natural Language Inference (NLI). To assess the performance and generalization ability of our twostage framework, we also evaluate CXRFEScore using the 3955 reports and associated tag annotations of the IU X-ray dataset (Demner-Fushman et al., 2015). It is important to highlight that radiologists or doctors have partially or fully annotated these datasets, which adds significant value to training and evaluation. We direct the reader to Appendix A.3 for further details on these datasets
and their utilization in our experiments.

heading: 3.4.1 LLM-assisted data augmentations for training supervision
text: As mentioned in Section 3.2, we use six distinct tasks to train CXRFE. For certain tasks, we leverage ChatGPT to generate additional training data. We elaborate in depth on these aspects in Appendix A.3, so here we only offer a concise overview.
For the triplet loss task (detailed in Section 3.4.2), some triplets incorporate paraphrased facts, which we generate using ChatGPT (an example is depicted in Figure 15). Similarly, we employ ChatGPT to produce challenging triplets, as indicated by the prompt in Figure 21.
In classification-related tasks, each fact is annotated with a JSON metadata object using ChatGPT. This object encompasses fields such as anatomical location, detailed observation, short observation, category, health status, and comparison status. The respective prompt for this process is shown in Figure 17. To refine the "comparison status" field, we use another prompt showcased in Figure 18. Also, we employ ChatGPT to label according to the scheme of the Chest ImaGenome dataset, producing additional observation and anatomical location labels, as shown in Figures 19 and 20.
For the task of natural language inference, we extensively utilize GPT-4 to generate training examples with distinct prompts (refer to Figures 22, 23, 24, 25, and 26).

heading: 3.4.2 Triplet Sampling
text: CXRFE is trained to generate sentence embeddings that cluster semantically similar sentences in the embedding space through a triplet ranking task with binary cross-entropy loss. This approach uses a dataset of triplets, each one with an anchor, a positive sample (akin to the anchor), and a negative one. The difference in similarities is computed as
∆sim(a, p, n) = sim(a, p) − sim(a, n) from their embeddings’ dot product. By minimizing the binary cross-entropy loss, the encoder ensures closely related sentences are nearer and unrelated ones are more distant in the embedding space.
We define six triplet sampling rules to guide the selection process. Rule 1 prioritizes paraphrases generated with ChatGPT. Rule 2 involves sampling triplets based on the consensus of BioVIL-T and Levenshtein distance, with the anchor and positive sample sharing the same health status. Rule 3 ensures proximity between short observations, detailed observations, and original facts, along with their paraphrases. Rule 4 samples triplets based on Chest ImaGenome labels, ensuring that the anchor and positive sample share at least one label and that BioVIL-T and Levenshtein distance agree. Rule 5 ranks triplets according to the overlap of entities and relations from RadGraph. Rule 6 includes hard triplets generated by ChatGPT. For each rule, we sampled around 3 to 4 million training triplets and 1,000 each for validation and testing.
These rules encapsulate specific intuitions and heuristics regarding the ranking of sentence embeddings within the semantic space. The design of these sampling rules and the construction of the triplets dataset involve several technical details, which are elaborated upon in Appendix A.4.

heading: 3.4.3 Natural Language Inference
text: Natural Language Inference (NLI) aims to classify the relationship between a premise and a hypothesis into one of three categories: "entailment", "neutral", or "contradiction". For instance, consider a premise stating “There are no evident signs of pleural effusion”, while a hypothesis asserts “There are evident signs of pleural effusion”. Despite their structural and lexical similarities, these sentences express contradictory meanings. A robust
sentence embedding should be able to distinguish between sentences conveying contradictory diagnoses. Thus, our objective in leveraging NLI during training is to refine sentence embeddings to discern such subtle distinctions accurately.
All MedNLI splits (Romanov and Shivade, 2018) are used for training, amounting to 14,049 annotated sentence pairs. Radiology-specific datasets include RadNLI (Miura et al., 2021) with 960 pairs and MS-CXR-T (Bannur et al., 2023), an evaluation set with 361 pairs emphasizing condition evolution over time. Given the dearth of training data, we build a custom NLI dataset combining MedNLI (14,049), RadNLI development set (480) and GPT4 generated examples (154,498), resulting in a total of 169,025 pairs (entailment: 25,175, neutral: 44,729, contradiction: 99,121), while RadNLI test set (480) and MS-CXR-T (361) are set apart for evaluation.

heading: 4 Experimental Results
text: Our experiments evaluate different versions of CXRFE, each defined by a subset of the six tasks (T, C, NLI, EC, ER, SD) outlined in Section 3.2. This results in a total of 64 potential combinations. However, for the sake of simplicity, we heuristically assess only 11 combinations. Further details on our rationale are provided in Appendix A.3.
Triplet Ranking. We evaluate CXRFE and mul-
tiple baselines from the literature on triplet ranking accuracy, using a separate test set of 1000 triplets per rule sampled according to the sampling rules detailed in Section 3.4.2. The left side of Table 1 presents these results. Notably, all different versions of CXRFE outperform all the baselines in triplet rules where ChatGPT is heavily involved, namely, paraphrases (R1(o), R1(a), R3) and hard triplets (R6). The hard triplets are especially challenging for the baselines, with BioViL-T only achieving 0.765 accuracy (row 6), while the best performing version of CXRFE achieves 0.952 (row 17). In addition to triplet loss (T), which is key to learning an embedding for these rules, we notice that sentence decoding (SD) and classification (C) appear to be helpful auxiliary tasks since most of the best scores are achieved by variants that include them (rows 8, 10, 15, 17).
Sentence Ranking. To complement the triplet ranking evaluation, which is based on heuristic sampling rules, we conduct a sentence ranking evaluation using 2,412 carefully annotated sentences provided by radiologists from the gold standard of the Chest ImaGenome dataset (Wu et al., 2021). These sentences are annotated with a vocabulary of 70 observations (yes (1), no (0), or omitted (- 1)), and 38 anatomical locations (mentioned (1) or unmentioned (0)), resulting in a discrete vector of size 108 for each sentence.
We report the results of this evaluation on the right side of Table 1. In this evaluation, each sentence serves as a query, against which all other sentences are ranked based on the cosine similarity of their embeddings. AUC, acc@k, and cont@k are computed by comparing the labels of the query sentence with those of each ranked sentence. AUC requires defining which sentences are relevant (1) or irrelevant (0) for the query. For two sentences a and b, a is deemed relevant for b if a’s labels logically entail b’s or vice versa; otherwise, they are considered irrelevant. acc@k represents the mean average accuracy up to the kth sentence in the ranking, while cont@k represents the mean number of sentences contradicting the query up to the kth sentence, by having contradictory values in at least one observation (1 vs. 0).
CheXbert (row 4) consistently performs the strongest among the baselines, likely due to its training in a similar multi-label classification task. CXRFE achieves the best overall performance. Notably, the T+C variant (row 8), combining triplet loss (T) and classification (C), attains the highest AUC and accuracy, while T+EC+NLI (row 13), containing tasks designed for pulling contradictory sentences apart, yields the smallest cont@k scores (where smaller is better). Rows 14-17 represent intermediate points between these two extremes.
NLI. Table 2 presents the NLI results using cosine similarity between sentence vectors, following a methodology akin to Bannur et al. (2023). This methodology specifically focuses on entailment

heading: ID Text Model Accuracy
text: and contradiction pairs, aiming to assess the efficacy of a text embedding in distinguishing between the two given a similarity threshold. We present results across three datasets: our NLI custom dataset mentioned in Section 3.4.3, the RadNLI test set, and MS-CXR-T. Our reported results are based on thresholds fine-tuned in the NLI custom dataset, alongside upper bounds obtained by tuning thresholds within the same data used for evaluation.
Notably, employing the quadruplet entailment/contradiction loss (EC) and natural language inference (NLI) (rows 11 to 17) is essential to achieve high performance, significantly outperforming all baselines. In contrast, variants lacking EC and NLI (rows 7 to 10) exhibit a weaker result.
Among the baselines, CheXbert (row 4) demonstrates superior performance on RadNLI, while BioVil-T (row 6) is the clear victor on MS-CXR-T. However, all baselines struggle considerably in our NLI custom dataset and are outperformed across all three datasets by variants 11-17 of CXRFE.
Additionally, Table 3 presents the accuracy achieved on the RadNLI test set in the context of the typical 3-class classification task encompassing entailment, contradiction, and neutral classes. In this evaluation, we exclusively assess variants of CXRFE equipped with an NLI classification head (rows 9-15). For insights into implementing NLI classification, please consult Figure 7.
Within the existing literature, the strongest baseline identified is DoT5 (Liu et al., 2023a) (82.1), employing a sophisticated sequence-to-sequence approach based on T5. Furthermore, we conducted evaluations on GPT-4 (rows 5-6) and Meta-Llama3-8B (AI@Meta, 2024) (rows 7-8) utilizing two distinct prompts: a simple prompt (Figure 24) and a prompt with Chain-of-Thought (CoT) + exam-
ples (Figure 25). Notably, the second prompt led to a significant performance boost for GPT-4 (from 82.3 to 89.0), whereas Meta-Llama-3-8B, an opensource LLM from Meta, only experienced a moderate improvement (from 58.1 to 61.5), with very low accuracies overall. Consequently, GPT-4 with the second prompt was selected as our "oracle" for generating additional training data (more details on this in Appendix A.3).
Most versions of CXRFE showed superior performance compared to the baselines. Surprisingly, a version fine-tuned explicitly for NLI (row 15) even outperforms GPT-4 with CoT (row 6) by a narrow margin (89.8).
CXRFEScore vs. existing metrics. To assess the quality of our proposed metric, we conduct an evaluation of CXRFEScore alongside multiple metrics from the literature, as shown in Table 4. This assessment encompasses four settings: (1) a sentence ranking evaluation using 2412 sentences, (2) a report ranking evaluation with 500 reports, both sourced from the gold standard of Chest Im-
aGenome, (3) a report ranking evaluation with 3955 reports from the IU X-ray dataset (DemnerFushman et al., 2015) leveraging the manual and automatic tags associated with each report, and (4) a natural language inference evaluation utilizing entailment (336) and contradiction (424) pairs from RadNLI and MS-CXR-T. Note that all these datasets are annotated by radiologists, thus serving as gold standards for metric comparison.
Among the baseline metrics, RadGraph F1 (rows 10-11) emerges as one of the most promising based on its performance on Chest ImaGenome Gold, which aligns with the findings of Yu et al. (2022). However, BERTScore’s results (row 5) on Chest ImaGenome Gold are quite similar, achieving the highest AUC among the baselines (0.840). Additionally, BERTScore achieves the highest Jaccard index scores on IU X-ray among the baselines. Notably, CheXbert (rows 8-9), closely followed by CheXpert (rows 6-7), shows the fewest contradictions on Chest ImaGenome Gold.
All baseline metrics, however, are surpassed by CXRFEScore (rows 12-15) in all the evaluation metrics. A particularly striking observation is that the baseline metrics struggle significantly to differentiate between entailed and contradictory sentences, as indicated by the AUC results in the last column of Table 4. RadGraph F1 Full achieves an AUC of only 0.610, whereas the best version of CXRFEScore (row 15) achieves an AUC of 0.938. This suggests that current metrics assign elevated scores to pairs of sentences with contradictory semantics, highlighting the necessity for improved metrics to discern these subtleties—precisely what CXRFEScore is designed to accomplish.
Similarly, CXRFEScore outperforms all the baselines on IU X-ray, a dataset not used to develop the metric. This provides valuable evidence of the metric’s ability to generalize to radiology reports from a different institution.
We provide additional details and results about these metrics in Appendix A.5.
Fact Extraction Quality. To evaluate the quality of the fact extraction stage, we interpret these facts as open-vocabulary labels and compare them against three existing radiology report label extraction methods: CheXpert labeler (Irvin et al., 2019), CheXbert (Smit et al., 2020), and Chest ImaGenome (Wu et al., 2021). For Chest ImaGenome, we use the labels from the dataset’s scene graphs, as the original NLP algorithm is not publicly available. For fact extraction, we compare T5-small,
fine-tuned specifically for this task, against GPT-4 and Meta-Llama-3-8B. The latter two models use the prompt shown in Figure 14. Our evaluation protocol involves the following: for each MIMICCXR test set report and label extraction method, labels are extracted, converted into a report, and evaluated against the original report by several metrics. We adopt the templates suggested by Pino et al. (2021) for CheXpert labeler and CheXbert, while for Chest ImaGenome, we utilize basic templates such as “(no) {observation} in {anatomical location}”. For fact extraction, we simply concatenate the facts. These template-based reports are illustrated in Tables 7 and 8.
Table 5 presents these results. Notably, factbased reports generated by T5 (row 4) achieve most of the top scores, even slightly outperforming GPT4 and Meta-Llama-3-8B, demonstrating the efficacy of the fact extraction process. Interestingly, CXRFEScore suggests that Chest ImaGenome outperforms both the CheXpert labeler and CheXbert, which is reasonable given Chest ImaGenome’s broader range of labels, although the improvement is rather marginal. However, it is evident that all three baseline methods fall short of fully capturing the factual information within the reports, likely due to their rigid annotation rules. This shortcoming is highlighted by their results in CXRFEScore, RadGraph F1, BERTScore, and most of the other metrics, compared to the fact extraction methods.

heading: 5 Conclusions & Future Work
text: In this work, we present a novel two-stage framework for extracting and encoding factual information from radiology reports. The first stage, fact extraction, uses ChatGPT and T5 to extract factual statements. The second stage, fact encoding, introduces CXRFE, a specialized variant of CXRBERT, fine-tuned through multitask learning by incorporating tasks that support representation improvement. Our system’s effectiveness is validated through comprehensive evaluations. Additionally, we introduce CXRFEScore, a novel metric for evaluating radiology text generation, leveraging our two-stage system. We anticipate that our work will stimulate further research in enhanced fact extraction and representation, LLM utilization, advanced training methodologies, and improved evaluation metrics. For future work, we aim to expand our framework to integrate visual modality, focusing on image-based fact detection and visual grounding.

heading: Limitations
text: Our study acknowledges several limitations and highlights areas for improvement. First, more expert evaluations, particularly from radiologists, are needed to refine the use of large language models (LLMs) in radiology. Although we extensively utilized publicly available gold standards, such as those from the Chest ImaGenome dataset, RadNLI, and MS-CXR-T, there remains room for improvement. For instance, involving radiologists in the prompt engineering process and developing more rigorous evaluation protocols are two strategies we believe will enhance the evaluation and utilization of LLMs for radiological text.
We also see potential in designing better triplet sampling heuristics, especially with input from radiologists. Optimizing LLM prompts for triplet sampling and incorporating more advanced auxiliary embeddings could further enhance our approach.
Furthermore, while our study focuses on textonly analysis, we recognize the importance of integrating visual data, such as chest X-ray images, into a multimodal framework. Devising a training protocol that effectively combines supervision from both images and text is an area of potential improvement for future work.
In this work, we limited our experiments to the sections "findings," "impression," and similar headings providing factual statements about the imaging exam. However, other sections, such as "comparison," "indication," and "history," were left out of the analysis, yet they can provide deeper insights into patient information and context. Investigating how this broader information can be extracted and encoded to enhance downstream applications is another avenue for future exploration and potential improvement.
Lastly, we acknowledge that our fact extraction algorithm may be limited due to its reliance on the sent_tokenize function of the NLTK library, which we use to obtain a preliminary division of the report into coarse sentences (before fact extraction). This method could falter when a fact spans multiple sentences connected through co-reference. While such occurrences are relatively uncommon in our observations, a deeper exploration of this linguistic aspect could guide the development of a more refined fact extraction mechanism that overcomes this challenge.

heading: Acknowledgements
text: This work was funded by the Chilean National Agency for Research and Development (ANID), including Instituto Milenio en Ingeniería e Inteligencia Artificial para la Salud (iHEALTH) ICN2021_004; Centro Nacional de Inteligencia Artificial (CENIA) FB210017; Fondecyt regular 1231724; Fondecyt 1221425; and the ANID Scholarship Program / Doctorado Becas Chile / 2019 - 21191569. Additionally, Pablo was supported by the National Institutes of Health (NIH) grant 1R01AG067396. We are grateful for the support from all funding sources mentioned above.

heading: A Appendix

heading: A.1 Additional Related Work
text: Evaluation of Factual Correctness in Radiology Text Generation. One important area of application motivating this work is the evaluation of
factual correctness in systems that generate radiological text, usually from input medical imaging. Recent research emphasizes enhancing the accuracy of generated facts in applications like report generation (Miura et al., 2021; Delbrouck et al., 2022; Pino et al., 2020, 2021) and summarization (Zhang et al., 2020b; Delbrouck et al., 2023). Pino et al. (2020) conducted an evaluation of several trivial report generation baselines using established metrics such as BLEU (2002), ROUGE-L (2004), and CIDEr-D (2015). They achieved results comparable to state-of-the-art papers at the time. However, when assessed using the CheXpert labeler (Irvin et al., 2019), a domain-specific NLP tool designed to detect 13 findings, the performance was notably poor, underscoring the urgent necessity for standardizing improved evaluation metrics among researchers. More recently, Delbrouck et al. (2022) repurposed RadGraph’s entity and relation extraction model (Jain et al., 2021a) to create a factual correctness reward. This reward measures the overlap of entities and relations between real and generated reports, serving as the guiding signal to optimize a report generation model through reinforcement learning. Interestingly, their proposed reward aligns functionally with the RadGraph F1 metric introduced by Yu et al. (2022). Yu et al. conducted a study on metrics for radiology report generation, determining that RadGraph F1 and BLEU show the highest correlation with radiologists’ judgement. Recently, the RadSum23 challenge (Delbrouck et al., 2023) evaluated multimodal radiology report summarization quality using BLEU-4 (2002), ROUGE-L (2004), BERTScore (2020a), CheXbert F1 (2020), and RadGraph F1 (2021a).
Our work is highly relevant in this domain because of our development of the CXRFEScore metric. This metric leverages the strengths of both stages within our framework: fact extraction and encoding. As a result, CXRFEScore is specifically designed to assess the factual accuracy of generated radiological text against a reference text. Section 4 offers a comprehensive evaluation, showcasing the effectiveness of our two-stage system. This includes a comparison of CXRFEScore with commonly used metrics in the literature, with very favorable results.
LLMs in Medicine. Our work falls under the category of applications of LLMs to the medical domain. Specifically, in this work we make use of ChatGPT versions GPT-3.5 and GPT-4 through
OpenAI’s API3. Recent works have shown the effectiveness of ChatGPT applied to medical tasks. Most notably, recent work by Liu et al. (2023c) explored the boundaries of GPT-4 in radiology, evaluated in tasks such as classification, summarization, and natural language inference, with remarkable performance. Liu et al. (2023b) employed ChatGPT to generate short sentences with plausible symptoms of medical conditions for interpretable zero-shot medical image diagnosis. Adams et al. (2023) used GPT-4 to transform free-text radiology reports into structured templates, with remarkable results. GPT-4 is also known for having passed the bar exam (Katz et al., 2023). Inspired by these results, we make extensive use of ChatGPT to produce abundant annotations through diverse prompts.
Knowledge Distillation from LLMs. Our approach can also be viewed as a form of LLM knowledge distillation, where a large language model (LLM) "teacher" generates annotations for training a more compact "student" model. Shi et al. (2023) illustrated this by using ChatGPT to extract knowledge graphs from text to train a smaller model for text classification. Similarly, Gu et al. (2023) applied this concept in the biomedical field, distilling knowledge from GPT-3.5 for adverse drug event extraction with student models like PubMedBERT and BioGPT. In line with these works, our research presents a form of knowledge distillation as we fine-tune T5 (student) using annotations generated by ChatGPT (teacher). This process trains an imitator that is more cost-effective and easier to run on our machines. We apply this method to the tasks of fact extraction from reports and the generation of metadata and labels for each fact. Additionally, we extensively use GPT-4 to produce high-quality NLI labels. Although we don’t use T5 in this case, these labels serve as part of the training supervision for CXRFE, which can also be considered a form of knowledge distillation.

heading: A.2 Fact Extraction Implementation Details
text: In our experiments, we processed the 227,827 radiology reports provided by the MIMIC-CXR dataset (Johnson et al., 2019b). To pinpoint relevant radiological sections in the MIMIC-CXR reports, such as Findings, Impression, and various other headings, we employed a combination of regular expressions and simple rules. These sections
3https://platform.openai.com/
were then segmented into sentences using NLTK’s sent_tokenize function, resulting in 677,694 unique sentences after processing the entire dataset. Subsequently, we extracted facts from each sentence.
Extracting factual statements from a sentence of a free-text radiology report using traditional approaches, such as regular expressions, handdesigned rules, and similar heuristics, poses significant challenges due to the complexity and diversity of vocabulary used by radiologists. A more promising alternative is to leverage the capabilities of powerful LLMs like ChatGPT to tackle this task. Table 6 presents several examples of facts extracted by GPT-4 from challenging sentences. This is achieved by providing the model with a specific set of instructions: (Refer to Figure 14 for a screenshot of OpenAI’s web interface displaying the same prompt.)
Relevant facts:
1. observations of abnormalities 2. observations of diseases 3. observations of strange visual patterns 4. observations of devices 5. observations of foreign bodies 6. observations of specific anatomical regions that look normal or healthy 7. absences of abnormalities (usually expressed with a negation) 8. comparisons with respect to a previous study (something changed or remained the same)
Task:
Given a sentence taken from a chest x-ray report, generate a JSON list of relevant facts. Each fact should be about one observation. If a sentence mentions multiple observations, each observation should be extracted as a separate fact. Each fact should include the anatomical location where it was observed. If multiple facts occur in the same location, repeat the location in each fact.
If no relevant facts are mentioned, return [] (an empty array).
Examples:
Opacity and density in the right lobe
[ "opacity in the right lobe", "density in the right lobe" ]
Lungs are well inflated without evidence of focal airspace consolidation to suggest pneumonia.
[ "well inflated lungs", "lungs without evidence of focal airspace consolidation", "lungs without evidence of pneumonia" ]
Taken together, compared with less than 1 hr earlier, the findings are suggestive of worsening of CHF, with new or significantly increased left greater right pleural effusions and underlying bibasilar collapse and/or consolidation, particularly on the left.
[ "worsening of CHF", "new or significantly increased left pleural effusions", "new or significantly increased right pleural effusions", "underlying bibasilar collapse on the left", "underlying consolidation on the left", ]
No acute cardiopulmonary abnormality
[ "no acute cardiopulmonary abnormality" ]
Given the relatively high cost of both GPT-4 and GPT-3.5 models, we opted to annotate a subset of the MIMIC-CXR sentences with these LLMs. To select this subset, we employed a two-pronged approach. Firstly, we ranked sentences based on the sum of the inverse frequency of their tokens, thus prioritizing longer and more complex sentences, which often include infrequent abnormalities, typos, and symbols. Secondly, to ensure diversity of topics, we clustered the sentences into 200 groups using K-Means, utilizing embedding representations obtained with BioViL-T, and sampled equally from each cluster in order of difficulty. This combined strategy allowed us to curate a subset that is both diverse and challenging. GPT-4-0613 extracted facts from 24,998 sentences, while GPT-3.5turbo-0613 processed 69,936. Subsequently, we trained T5-small for fact extraction using a total of 94,934 training examples, reserving 200 examples for validation. Once trained, T5-small annotated the remaining 582,760 sentences, yielding a total of 1,341,830 facts, including duplicates, of which 591,920 were unique after duplicate removal.

heading: A.3 CXRFE’s Tasks Details
text: CXRFE is a fine-tuned version of CXRBERT-specialized, accessible for download from https://huggingface.co/microsoft/ BiomedVLP-CXR-BERT-specialized. This fine-tuning process entails multi-task learning, incorporating six distinct tasks: triplet loss for sentence ranking (T), sentence classification (C), sentence decoding (SD), natural language inference (NLI), quadruplet loss for enforcing separation between entailment and contradiction pairs (EC), and entity and relation extraction (ER). While there exist a total of 64 possible task combinations, we focus on 11 combinations in our experiments, namely: T, T+C, T+ER, T+SD, T+EC, T+NLI, T+EC+NLI, T+C+EC+NLI, T+C+EC+NLI+ER, T+C+EC+NLI+SD, T+C+EC+NLI+ER+SD. This selection is guided by the following heuristics:
• Triplet loss (T) is always included, given our belief, as discussed in Appendix A.4, that the developed triplets dataset captures many desirable properties for learned sentence embeddings.
• We explore each combination of T with the other tasks individually: T+C, T+ER, T+SD, T+EC, T+NLI.
• Recognizing the complementary nature of EC and NLI in exploiting natural language inference data, we explore T+EC+NLI.
• According to our experimental results, both C and EC+NLI serve as effective auxiliary tasks; thus, we keep them fixed while varying other combinations of SD and ER: T+C+EC+NLI, T+C+EC+NLI+ER, T+C+EC+NLI+SD, T+C+EC+NLI+ER+SD.
Next, we delve into the implementation details of each task in our experiments.
Triplet loss for sentence ranking (T). One of the tasks we explore for model fine-tuning is sentence ranking via triplet loss. Figure 4 illustrates this task. Concretely, we forward 3 sentences (anchor, positive, negative) through CXR-BERTspecialized with weight sharing, obtaining three vectors a, b, and c each of dimension 128, and compute ∆sim(a, p, n) = a · p− a · n. This is serves as the input logit for a binary cross-entropy loss.
We provide details on the construction of the triplets dataset used in this task in Section A.4.
Sentence classification tasks (C). A second task group is classification tasks (Figure 5). These include category (5 classes: anatomical finding, disease, technical assessment, tubes and lines and device), health status (4 classes: normal, abnormal, ambiguous, unknown), comparison status (15 classes, see Figure 18), Chest ImaGenome observations (74 classes, see Figure 19) and anatomical locations (38 classes, see Figure 20). Category, Health Status and Comparison Status are singlelabel multi-class classification tasks, whereas Chest ImaGenome observations and anatomical locations are multi-label binary classification tasks. Implementing these tasks require attaching fully con-
nected heads on top of CXR-BERT-specialized’s built-in projection layer in order to perform the classification.
The data for the classification tasks comes from two primary sources. The first source is the Chest ImaGenome dataset, which provides observation and anatomical location labels in scene graphs, conveniently linked to specific sentences from MIMICCXR reports. The second source is ChatGPT,
which we leverage to annotate the extracted facts with details such as category, health status, comparison status, observations, and anatomical locations. This process involves a teacher-student approach similar to our method for fact extraction: we use ChatGPT to annotate a subset of facts, incurring some monetary cost, with the prompts shown in Figures 17, 18, 19, and 20, and then fine-tune T5 to annotate the remaining facts at no additional cost.
Concretely, from the Chest ImaGenome scene graphs, we retrieved 556,111 sentences, each annotated with observations (74 classes) and anatomical locations (38 classes). In addition, facts were annotated as follows:
• 5,000 facts were annotated with observations by GPT-4-0613. • 84,708 facts were annotated with observations by GPT-3.5-turbo-0613. • 2,816,982 facts were annotated with observations by T5-small fine-tuned with the teacherstudent approach. These include facts extracted from reports plus additional facts obtained via paraphrases. (For more details on
paraphrases, see Section A.4 on the construction of the triplets dataset.)
Similarly, anatomical locations were annotated as follows:
• 72,400 facts were annotated with anatomical locations by GPT-3.5-turbo-0613. • 2,598,778 facts were annotated with anatomical locations by T5-small fine-tuned.
For other classification tasks (Category, Health Status, Comparison Status), we generated a JSON object with metadata from each fact using the prompt shown in Figure 17:
• 59,921 facts were annotated by GPT-3.5-turbo-0613. • 535,959 facts were annotated by T5-small fine-tuned with the teacher-student approach.
Sentence decoding (SD). Another task is sentence decoding (Figure 6). We attach a lightweight, shallow Transformer Decoder to CXR-BERTspecialized’s projection layer in order to generate back the original sentence. This can be viewed a sort of text autoenconder, forcing the projection layer to capture as much information as possible of the input sentence to facilitate the reconstruction of the sentence by the Transformer Decoder. We use a Transfomer Decoder with embedding, hidden and feedforward dimension 256, only one self-attention head and only one layer.
Because this task relies on self-supervision and does not require specialized annotations, any sentence or fact can serve as a training instance.
Natural language inference (NLI). For NLI, we adopt an approach similar to that of SBERT (Reimers and Gurevych, 2019), by concatenating the embeddings of the premise, hypothesis and their element-wise multiplication, followed by a fully connected layer and a softmax layer for NLI classification (see Figure 7).
As discussed in Section 3.4.3, our NLI dataset was compiled by merging data from MedNLI (14,049 pairs), the RadNLI development set (480 pairs), and examples generated by GPT-4 (154,498 pairs), resulting in a comprehensive collection of 169,025 pairs. These consist of 25,175 entailment pairs, 44,729 neutral pairs, and 99,121 contradiction pairs. Additionally, the RadNLI test set (480 pairs) and MS-CXR-T (361 pairs) were reserved for evaluation purposes. To generate NLI examples using GPT-4, we employed four distinct prompts:
• A prompt that generates entailment, neutral, and contradiction sentences from a reference sentence (Figure 22). • A prompt that aims to produce examples analogous to a given NLI reference example (Figure 23). • A prompt that predicts the correct label for a given premise and hypothesis, incorporating Chain-of-Thought (CoT) reasoning and examples (Figure 25). A simpler version of this prompt, which only requests the correct label, was also considered (Figure 24). However, as indicated in Table 3, CoT is crucial for achieving significantly more accurate predictions. • A prompt designed to generate contradictory sentences relative to a reference sentence (Figure 26).
Quadruplet loss: enforcing separation between entailment and contradiction pairs (EC). The next task is what we refer to as entailment/contradiction quadruplet loss (Figure 8). The goal of this task is to promote a generalized separation of entailment and contradiction sentence pairs in the latent space, by randomly sampling entailment and contradiction pairs and requiring that the entailment pair have greater similarity than the contradiction pair. This loss was a key contributor to achieving the state-of-the-art results presented in Table 2.
Since this task complements the standard NLI task, it utilizes the same dataset but excludes neutral pairs, focusing solely on entailment and contradiction pairs.
Entity and relation extraction (ER). Lastly, for entity and relation extraction we augment CXR-BERT-specialized with the layers proposed by SpERT (Eberts and Ulges, 2020), as illustrated in Figure 9. This adaptation was relatively straightforward, since the authors of SpERT released an implementation (https://github.com/ lavis-nlp/spert/) that is compatible with Hug-
gingface models like CXR-BERT-specialized. To implement this task, we utilize the gold data provided by the RadGraph dataset (Jain et al., 2021a). This dataset comprises 500 MIMIC-CXR radiology reports, annotated with an entity-andrelation schema by board-certified radiologists. Additionally, a test set containing 50 MIMIC-CXR and 50 CheXpert reports, annotated in the same manner, is included. We use all of this data for training.

heading: A.4 Triplets Dataset Construction Details
text: CXRFE is trained to generate sentence embeddings that organize semantically similar sentences into clusters within the embedding space via a triplet ranking task. This task involves a dataset comprising triplets (a, p, n), where a represents an anchor sentence, p is a positive sentence, and n denotes a negative sentence. The objective is to train a text encoder in a manner that ensures sim(a, p) > sim(a, n) holds true for each triplet, with sim(x, y) computed as the cosine similarity between sentence embeddings (or dot product if they are already normalized).
The rationale behind selecting triplet loss lies in its versatility, offering flexibility in terms of desired properties for the learned embedding function.
For instance, if a and p denote identical medical observations using different vocabulary, while n presents an unrelated observation, it’s logical to enforce sim(a, p) > sim(a, n). Similarly, if there’s substantial topic overlap between a and p, whereas n is largely unrelated or contradictory to a, it’s reasonable to aim for sim(a, p) > sim(a, n) to hold true.
To guide a text encoder towards learning sentence embeddings consistent with triplets, we can define the difference in similarities as ∆sim(a, p, n) = sim(a, p) − sim(a, n). Then, a straightforward approach is to utilize sim(a, p, n) as the input for a binary cross-entropy loss, where the ground-truth label is consistently set to 1.
Notation. To define our triplet sampling heuristics, we use the notation E(x) for the embedding of sentence x, cos(E(x), E(y)) for the cosine similarity between embeddings of x and y, lev(x, y) for the Levenshtein string distance between them, and levsim(x, y) = 1 - lev(x, y) / max(len(x), len(y)). c(x) indicates the cluster id for sentence x after running a clustering algorithm like K-Means on the sentence embeddings. This requires having an auxiliary text encoder capable of producing these auxiliary embeddings and clusters. Specifically, we use BioViL-T (Bannur et al., 2023), a stateof-the-art BERT-based model for radiological text, available on Huggingface4.
With this, we sample triplets based on the following heuristics:
Rule 1: Rank paraphrases highly. ∆sim(a, p, n) > 0 if p is a paraphrase of a generated by ChatGPT and n is any other sentence (unless cos(E(a), E(p)) < cos(E(a), E(n)) and lev(a, p) > lev(a, n)). To generate paraphrases, we employ the prompts shown in Figures 15 and 16, one for paraphrasing facts and another for paraphrasing anatomical locations. We decided to paraphrase anatomical locations too in order to strengthen the model’s understanding of their vocabulary. As a reminder, the anatomical locations are obtained from facts as part of the metadata generated with the prompt shown in Figure 17.
Rule 2: Sample triplets according to the consensus of E and lev, while anchor and positive share the same health status. ∆sim(a, p, n) > 0 if HS(a) = HS(p), c(p) = c(a), c(p) ̸= c(n),
4https://huggingface.co/microsoft/ BiomedVLP-BioViL-T
cos(E(a), E(p)) > cos(E(a), E(n)) + margincos and levsim(a, p) > levsim(a, n) + marginlev. This rule is very heuristic, and can be interpreted as a sort of mixture of experts. Essentially, we are looking for multiple signals to agree, increasing the likelihood that the resulting triplet passing those tests is in fact a high quality triplet. Levenshtein distance is useful to capture the notion of string similarity, while BioViL-T captures more the semantics (to some extent). The health status condition is adding an additional constraint. As a reminder, the health status is also obtained with the prompt of Figure 17.
Rule 3: Short observation, detailed observation and the original fact (and their paraphrases) should be close to each other. Given a fact f , ∆sim(a, p, n) > 0 if a and p ∈ S(f ), n /∈ S(f ) and c(a) ̸= c(n) (unless cos(E(a), E(p)) < cos(E(a), E(n)) and lev(a, p) > lev(a, n)). Here, S(f ) stands for the union of f , its detailed observation, its short observation and all the paraphrases (if any) generated for all of them with ChatGPT. The intuition here is that they are all closely related semantically, as they are all derived from the same fact (see Figure 17).
Rule 4: Sample triplets according to Chest ImaGenome labels. ∆sim(a, p, n) > 0 if CIGL(a) ∩ CIGL(p) ̸= ∅, CIGL(a) ∩ CIGL(n) = ∅, CIGL(p) ∩ CIGL(n) = ∅, and if (cos(E(a), E(p)) > cos(E(a), E(n)) AND levsim(a, p) > levsim(a, n)). Here, CIGL(x) stands for the set of Chest ImaGenome labels of the sentence x. With this rule, our aim is to heuristically utilize the Chest ImaGenome labels to identify semantically similar sentences that warrant clustering together. However, we enhance this approach by incorporating signals from Levenshtein distance and BioViL-T to bolster our confidence in the triplet quality.
Rule 5: Rank triplets according to the overlap of entities and relations from RadGraph. ∆sim(a, p, n) > 0 if c(a) = c(p), c(a) ̸= c(n), and J(RG(a), RG(p)) > J(RG(a), RG(n)) + marginRG. Here, RG(x) stands for the set of RadGraph entities and relations for the sentence x, and J for Jaccard similarity. Following the same idea of the previous rule, we seek to utilize the entities and relations provided by the RadGraph dataset as valuable cues for identifying semantically similar sentences that ought to be clustered together in the embedding space.
Rule 6: Hard triplets generated by ChatGPT. ∆sim(a, p, n) > 0 if (a, p, n) is a hard triplet gen-
erated by ChatGPT. The intuition behind this rule is very simple: we want to leverage ChatGPT’s remarkable skills to produce challenging triplets, requiring a good understanding of the text to be ranked correctly. Figure 21 shows the prompt used to generate these triplets along with an example.
For each rule, we sample approximately 3 to 4 million training triplets, along with 1,000 triplets for validation and 1,000 triplets for testing.

heading: A.5 Additional Metric Evaluation Details and Results
text: We conduct a thorough comparison involving CXRFEScore against several established metrics, including BLEU (Papineni et al., 2002), ROUGE-L (Lin, 2004), METEOR (Banerjee and Lavie, 2005), CIDEr-D (Vedantam et al., 2015), BERTScore (Zhang et al., 2020a), CheXpert labeler (Irvin et al., 2019), CheXbert (Smit et al., 2020), and RadGraph F1 (Delbrouck et al., 2022). The last three are considered domain-specific metrics, tailored to the radiology domain, while the others serve as generalpurpose evaluation metrics.
Both the CheXpert labeler and CheXbert output a 14-dimensional discrete vector representing 13 observations along with a label indicating "no findings". The values in this vector denote presence (1), absence (0), uncertainty (-1), and unknown (- 2). We binarize this vector, treating both presence and uncertainty as 1, and absence and unknown as 0. This enables the computation of CheXpert F1, CheXbert F1, CheXpert Accuracy, and CheXbert Accuracy.
Regarding the RadGraph F1 metric, drawing inspiration from the methodology of Delbrouck et al. (2022), we employ the pretrained entity and relation extraction model provided within the dataset. This model has been conveniently made accessible as an installable package (https://pypi.org/ project/radgraph/). Specifically, we assess the RadGraph F1 Partial variant included in the package, as it was the officially designated variant for the First Shared Task on Clinical Text Generation: RRG24 (Xu et al., 2024). Additionally, we explore another variant, which we term RadGraph F1 Full. This variant is based on the underlying model of the package. We achieve this by creating a "bag" comprising entities, relations without type, and relations with type, and then computing the F1 score between the "bag" of a referenced report and that of a generated report.
In addition to the results that were already pre-
sented in Table 4 (Section 4), in this appendix we include the plots shown in Figures 10, 11, 12, 13.

heading: A.6 Hardware and Other Implementation Details
text: All of our experiments are implemented using Python 3.10.10 with PyTorch version 1.13.1+cu117 (Paszke et al., 2017). All experiments are conducted on a computing node equipped with a
20-core Intel(R) Core(TM) i9-9900X CPU @ 3.50GHz, three NVIDIA GPUs - two GeForce RTX 2080 Ti with 11GB memory and one GeForce RTX 3090 with 24GB memory. The system is complemented by 125GB of RAM.
We implement multitask learning for CXRFE using interleaved dataloaders, multiple model for-
ward passes, and multiple gradient accumulation steps. Specifically, our model features distinct forward functions for each task, with each task assigned its own dataloader. These dataloaders are interleaved according to weights that determine the sampling frequency for each task. To ensure all tasks contribute to the gradients during training, we
employ enough gradient accumulation steps so that each task’s batch is sampled at least once before performing backpropagation.
We use the AdamW optimizer (Loshchilov and Hutter, 2019) with a cyclic exponential learning rate that varies from 8e-5 to 1e-6 over 8 epochs. Here, an epoch consists of roughly 800 batches. Typically, our experiments run for 12-18 hours, after which we observe no significant gains in validation metrics.
A.7 ChatGPT prompts

heading: Future Work
text: Devising a training protocol that effectively combines supervision from both images and text is an area of potential improvement for future work. In this work, we limited our experiments to the sections "findings," "impression," and similar headings providing factual statements about the imaging exam. However, other sections, such as "comparison," "indication," and "history," were left out of the analysis, yet they can provide deeper insights into patient information and context. Investigating how this broader information can be extracted and encoded to enhance downstream applications is another avenue for future exploration and potential improvement. Lastly, we acknowledge that our fact extraction algorithm may be limited due to its reliance on the sent_tokenize function of the NLTK library, which we use to obtain a preliminary division of the report into coarse sentences (before fact extraction). This method could falter when a fact spans multiple sentences connected through co-reference. While such occurrences are relatively uncommon in our observations, a deeper exploration of this linguistic aspect could guide the development of a more refined fact extraction mechanism that overcomes this challenge