Boosting Radiology Report Generation by Infusing Comparison Prior
Year: 2023

Abstract:
Recent transformer-based models have made significant strides in generating radiology reports from chest X-ray images. However, a prominent challenge remains: these models often lack prior knowledge, resulting in the generation of synthetic reports that mistakenly reference non-existent prior exams. This discrepancy can be attributed to a knowledge gap between radiologists and the generation models. While radiologists possess patientspecific prior information, the models solely receive X-ray images at a specific time point. To tackle this issue, we propose a novel approach that leverages a rule-based labeler to extract comparison prior information from radiology reports. This extracted comparison prior is then seamlessly integrated into stateof-the-art transformer-based models, enabling them to produce more realistic and comprehensive reports. Our method is evaluated on English report datasets, such as IU X-ray and MIMIC-CXR. The results demonstrate that our approach surpasses baseline models in terms of natural language generation metrics. Notably, our model generates reports that are free from false references to non-existent prior exams, setting it apart from previous models. By addressing this limitation, our approach represents a significant step towards bridging the gap between radiologists and generation models in the domain of medical report generation.

text: The 22nd Workshop on Biomedical Natural Language Processing and BioNLP Shared Tasks, pages 50–61 July 13, 2023 ©2023 Association for Computational Linguistics

heading: 1 Introduction
text: The analysis of radiology images and the subsequent writing of medical reports are crucial tasks performed during the diagnostic process (Suetens, 2017; Krupinski, 2010). However, producing a radiology report is a labor-intensive and timeconsuming task for radiologists, requiring years of training to accurately identify and describe specific abnormalities in medical images (Brady, 2017; Arenson and Dunnick, 2006). Inspired by the success of image captioning models in deep learning,
numerous studies have emerged proposing various models for automated radiology report generation, specifically focusing on chest X-ray images (Yuan et al., 2019; Li et al., 2019; Xue et al., 2018; Jing et al., 2017; Liu et al., 2019a). The automated generation of reports holds the potential to alleviate the high workload of radiologists and expedite the diagnostic process by providing preliminary reports that include useful keywords or observations (Johnson et al., 2019; Chen et al., 2020).
Despite the relative success of recent approaches in generating radiology reports from chest X-ray images (Endo et al., 2021; Johnson et al., 2019; Chen et al., 2020; Miura et al., 2020; RamirezAlonso et al., 2022; Nooralahzadeh et al., 2021), a crucial challenge remains unaddressed in these studies: the need to provide models with appropriate prior knowledge, akin to what is available to radiologists. Specifically, radiologists are equipped with information about the existence of previous reports and X-ray images, enabling them to compare current exams with past ones, and assess the patient’s progress, deterioration, or improvement (Suetens, 2017; of Radiology , ESR). These medical reports often incorporate specific words or phrases for comparison, such as "compared to the previous exam," "in the interval," "referring to the prior X-ray," and so on. In this paper, we refer to these words or phrases as prior expressions, which are also present in general medical datasets such as MIMIC-CXR (Johnson et al., 2019) and IU X-ray (Demner-Fushman et al., 2016), widely utilized for training and evaluating report generation tasks. The inclusion of prior expressions in medical reports is vital for accurate reporting. However, models trained on medical report datasets often generate reports with inappropriate or misused prior expressions, leading to relatively lower performance metrics. The challenge lies in effectively incorporating and utilizing prior expressions within the model’s generation process, a crucial aspect yet to be fully
50
resolved.
In Table 1, we present a comparison between ground truth reports and reports generated by two recent models, R2Gen (Chen et al., 2020) and M2Tr (Cornia et al., 2020), focusing on the presence of prior expressions. It is evident that the synthetic reports contain inappropriate priors. For instance, the synthetic report generated by R2Gen in the first row includes a comparison with the previous exam, despite the absence of any prior information in the ground truth report. Similarly, the synthetic report produced by M2Tr in the second row includes the phrase "again noted," indicating the existence of a previous image, while the ground truth report lacks any prior expression.
These falsely referenced reports, which include prior expressions, tend to yield lower evaluation metrics. In Figure 1, we utilize a rule-based labeler (explained in Section 3.1) to classify synthetic reports into two categories: negative and positive. The negative class represents reports without prior expressions, while the positive class comprises reports with prior expressions. We then plot the distribution of BLEU-4 (Papineni et al., 2002) scores for each class, along with their respective mean and standard deviation. The results clearly demonstrate that the positive reports achieve lower scores compared to the negative reports. This observation suggests that the transformer-based models do not effectively leverage the generated prior expressions, underscoring the significance of properly
incorporating prior information during the training process.
Table 1 and Figure 1 highlight a critical issue with state-of-the-art (SOTA) models in learning and generating prior expressions, which can potentially confuse radiologists when utilizing these models for report writing. This issue stems from the fundamental disparity between how radiologists compose reports and how models generate them. Radiologists have access to not only prior patient information such as previous exams and medical history, but also the current X-ray images. In contrast, report generation models are only provided
with X-ray images at a specific moment. With limited prior information, it becomes challenging for the models to generate comprehensive and insightful reports comparable to those created by experts.
In our paper, we address this knowledge gap between generation models and radiologists by infusing prior information into existing models. Our aim is to reduce the disparity and enable the improved models to produce more informative and practical reports. Since existing datasets such as IU X-ray and MIMIC-CXR do not contain prior information (previous X-ray images), we adopt a data-driven approach by consulting experienced radiologists. Inspired by the CheXpert labeler (Irvin et al., 2019), we develop a rule-based labeler that extracts prior information by identifying specific patterns and keywords in radiology reports. Notably, the rule-based labeler focuses on comparison phrases that indicate whether a medical report corresponds to the first or subsequent exams for each patient.
The main contributions of our paper are as follows:
1. We collaborate with radiologists to develop a rule-based labeler that identifies specific keywords and patterns in medical reports related to comparisons.
2. To incorporate prior information into the models, we propose an enhanced transformerbased architecture. This approach is straightforward to implement and can be seamlessly integrated as a plug-in method into modern report generation models.
3. Through empirical evaluation on the IU X-ray and MIMIC-CXR datasets, we demonstrate that our model outperforms baseline models in terms of performance metrics.
4. Furthermore, we conduct a comprehensive analysis to confirm that our model no longer generates false references, addressing a significant limitation observed in previous approaches.

heading: 2 Related Work
text: Initial research (Bai and An, 2018; Liu et al., 2019b) in radiology report generation employed a basic encoder-decoder architecture, where an encoder extracted key features from medical images
and converted them into a latent vector, and a decoder generated the target text from the latent vector. Typically, CNN (LeCun et al., 2015) was used as the encoder, and LSTM (Hochreiter and Schmidhuber, 1997) was chosen as the decoder. Subsequently, visual attention mechanisms were introduced to highlight specific image features and generate more interpretable reports (Zhang et al., 2017; Jing et al., 2017; Wang et al., 2018; Yin et al., 2019; Yuan et al., 2019). Recent studies (Lovelace and Mortazavi, 2020; Chen et al., 2020; Nooralahzadeh et al., 2021; Miura et al., 2020) have explored more advanced architectures using transformers to produce more comprehensive and consistent medical reports.
Alternatively, generating medical reports can be approached as a retrieval task, as similar sentences and a specific writing format are often repeated in most reports. Reusing diagnostic text from visually similar X-ray images may result in more consistent and accurate reports compared to generating an entire report from scratch. Li et al. (2019) demonstrated the superiority of retrieval-based models, outperforming many encoder-decoder-based models. More recently, Endo et al. (2021) introduced a retrieval-based model called CXR-RePaiR, which incorporated contrastive language image pretraining (CLIP) (Radford et al., 2021) to calculate the similarity between text and image embeddings. CXR-RePaiR generates predictions by selecting the most aligned report from a large report corpus given a specific X-ray image. They achieved SOTA performance on their newly developed metrics.
Previous approaches have primarily focused on enhancing model performance through advancements in model architecture, while paying relatively less attention to the distinctive characteristics of radiology reports, particularly those involving comparisons. However, there are a few notable exceptions, such as the work by Ramesh et al. (2022), which specifically explores the impact of prior expressions in reports. The authors of that study introduced a novel dataset named MIMIC-PRO, in which they identified and modified reports that contained hallucinated references to non-existent prior exams. The hallucinated references can be seen as a concept analogous to the prior expressions discussed in our paper. Ramesh et al. (2022) suggested a BioBERT-based model that paraphrased or removed sentences referring to previous reports or images, arguing that these expressions confuse the
model and result in falsely referenced sentences. They collaborated with experts to create a "clean" MIMIC-CXR test dataset and compared models trained on MIMIC-CXR and MIMIC-PRO.
In contrast to Ramesh et al. (2022), we take a different approach to address the issue of comparison priors: we include prior information in the model and enable it to generate more comprehensive reports in an end-to-end fashion, rather than entirely removing comparison priors. Writing comparisons using prior expressions in radiology reports is unavoidable in the real medical field, and constructing a clean and accurate dataset from real reports is also a laborious task. Thus, our work focused on directly applying the comparison prior to existing models such as R2Gen and M2Tr.

heading: 3 Method
text: As observed in Table 1, even the best models to date struggle with unexpected prior expressions. To address this problem, we propose a two-step approach: (1) constructing a rule-based labeler to differentiate reports with and without prior expressions (Section 3.1), and (2) extending the transformer-based architectures (R2Gen and M2Tr) to incorporate the comparison prior as input (Section 3.2). In the first step, we introduce a rule-based labeler that detects specific comparison prior expressions and categorizes each report as either a first exam (negative) or a following exam (positive), based on its detection. We draw inspira-
tion from the negation and classification principles of the CheXpert labeler (Irvin et al., 2019) to design our novel labeler. In the subsequent stage, we integrate our prior label as input into the stateof-the-art architectures to generate more practical and comprehensive reports, and we compare the results with the baseline models. By providing our novel model with the comparison prior, which is typically communicated to radiologists in real diagnostic scenarios, we enable the generation of more comprehensive and consistent medical reports.

heading: 3.1 Rule-based Labeler
text: Our rule-based labeler follows the fundamental structure of the CheXpert labeler (Irvin et al., 2019), which detects the presence of 14 observations in radiology reports based on fixed rules devised by experts. Consequently, our labeler consists of three distinct stages: mention extraction, mention classification, and mention aggregation. The labeler takes the Finding section of radiology reports as input and generates a binary output (0 or 1). A negative label (0) indicates a report without prior expressions, while a positive label (1) signifies the presence of prior expressions.
Mention extraction A mention is defined as a specific keyword likely to be included in prior expressions, such as "previous", "prior", "preceding", "previously", "again", "comparison", "interval", "increase", "decrease", "enlarge", and so on. In this stage, the labeler extracts mentions from each report and marks them within each sentence. However, it is important to note that even if certain sentences contain the designated keywords, the existence of a prior expression cannot be confirmed at this step since those keywords might be used in other contexts. For instance, the word "comparison" can be used in a sentence like "with no comparison studies," indicating the absence of prior expressions.
Mention classification After extracting mentions in the first stage, our labeler determines whether each mention corresponds to predefined prior expressions. As similar expressions are frequently employed in reports to denote a comparison with previous exams, we can formalize the patterns of these prior expressions into several key phrases familiar to experienced radiologists. For example, the phrase "compared / similar to {mention}" confirms the presence of prior reports, where
"{mention}" represents keywords such as "previous", "preceding", and "prior". Similarly, "{mention} seen/identified/visualized/ ... /noted" constitutes a prior expression when "{mention}" pertains to keywords like "again" and "previously".
Mention aggregation In the final stage, the labeler combines the classified mentions and generates either a negative label (0) or a positive label (1), with negative indicating a report without prior expressions and positive denoting the presence of prior expressions. Examples of the labeler’s outputs can be seen in Table 2, and the numbers of negative and positive exams in the IU X-ray and MIMIC-CXR datasets are shown in Table 3.

heading: 3.2 Extending Model
text: In this section, we describe how we integrate the comparison prior into existing models, such as R2Gen and M2Tr, to generate more informative and comprehensive reports.
Generation Process The generation process of R2Gen and M2Tr can be illustrated as shown in Figure 2. It follows the following flow: input radiology images X → visual embedding V → latent representation L → output report Y . Initially, chest X-ray images X are provided as inputs to the visual extractor, where X consists of the frontal image Xf and the lateral image Xl, represented as X = {Xf , Xl}. The visual extractor generates the visual embedding V = {v1, v2, ..., vS}, which comprises patch features vs ∈ Rd, with d being the size of the feature vectors. Subsequently, V undergoes multiple transformer layers
in the encoder to obtain the latent representation L = {l1, l2, ..., lT }, where the latent feature vector is denoted as lt ∈ Rf . Finally, the decoder utilizes L to generate the final output report Y .
Infusing Comparison Prior The comparison prior P ∈ R is generated from our rule-based labeler and it denotes a negative (0) or positive (1) label. We intend to incorporate the comparison prior into the existing data pipeline in such a way that the addition of the prior does not change the architecture or add any additional weights to train. Otherwise, it will become hard to measure the effect of comparison prior to the generative models. As a result, we added prior P to both Visual Embedding V and Latent Representation L in the generation models shown in Figure 2. The encoder should be given the prior information so that it can generate an appropriate intermediate representation. Furthermore, we also add P on L since the knowledge of P could be weakened after deep transformer layers in the encoder. The decoder will generate the output report based on the latent representation conditioned on P . This whole process emulates the radiologists’ examination with prior exams. Therefore, our new visual embedding Vnew and new latent representation Lnew can be calculated as follows:
Vnew = V ⊕ P, Lnew = L⊕ P (1)
where ⊕ indicates element-wise summation. The strength of our method is that it is applicable to most existing transformer-based models and does not require an extra dataset or information.

heading: 4 Experiment
text: Architecture To extract visual features, we utilize pretrained Convolutional Neural Networks (CNNs) such as DenseNet121 (Huang et al., 2017) and ResNet121 (He et al., 2016). Through empirical evaluation, we find that DenseNet performs better for our generation task, and thus, we select it as our base visual extractor. We adopt the structure of Meshed-Memory Transformer (M2Tr) (Cornia et al., 2020) and Relational Memory-driven Transformer (R2Gen) (Chen et al., 2020) to construct our encoder and decoder.
Datasets We evaluate our proposed methods on two widely-used English datasets for medical report generation tasks: IU X-ray (Demner-Fushman et al., 2016) and MIMIC-CXR (Johnson et al., 2019). The IU X-ray dataset is a publicly available radiology dataset that consists of 7,470 chest X-ray images and 3,955 radiology reports. Each report is paired with one frontal view image and, optionally, one lateral view image. MIMIC-CXR is a large chest radiograph database comprising 473,057 chest X-ray images and 206,563 reports. We train our model using intact data pairs, which include two images (frontal and lateral) and one report (Findings section). The datasets are divided into train, validation, and test sets following the data split described in Chen et al. (2020).
Training Details We first generate the comparison prior for each report using a rule-based labeler. Then, we train our model with the two images and the comparison prior as inputs, and the medical report as the output. We employ the Adam optimizer with an initial learning rate of 0.00005 for the visual extractor and 0.0001 for the encoder-decoder
model. The learning rate gradually decreases at pre-defined steps. All experiments are conducted with 3 different seeds and a batch size of 16 on an "NVIDIA GeForce RTX 1080 Ti" GPU. Our code implementation is based on the publicly available codes from Chen et al. (2020) and Nooralahzadeh et al. (2021).
Evaluation Metrics We report general natural language generation (NLG) metrics, including BLEU (Papineni et al., 2002), CIDEr (Vedantam et al., 2015), and ROUGE-L (Lin, 2004). These metrics are commonly used to evaluate the quality of generated text. BLEU measures the n-gram overlap between the generated text and the reference text, while CIDEr is based on cosine similarity between word embeddings and considers both unigrams and multi-word phrases. ROUGE-L evaluates the longest common subsequence between the generated text and the reference text. Including these metrics enables a quantitative comparison of the generated reports with the ground truth and previous models, providing insights into the performance of the proposed approach.
Results In Table 4, we present the results of our proposed approach, which incorporates prior information into state-of-the-art NLG models, on two medical image report generation datasets: IU XRay and MIMIC-CXR. Our approach consistently outperforms the baselines across all NLG metrics, demonstrating its effectiveness in improving the quality of medical image reports generated by NLG models.
On the IU X-Ray dataset, our approach achieves an average improvement of 11.58% and 4.49% on all NLG metrics for R2Gen and M2Tr models, re-
spectively, compared to the baseline models. Notably, the CIDEr metric shows the highest improvement, with an increase of 31.46% for R2Gen and 7.00% for M2Tr. This suggests that, as measured by CIDEr, our approach generates more diverse and contextually relevant captions, which align better with human judgments of quality than other metrics.
For the MIMIC-CXR dataset, our approach improves the R2Gen and M2Tr models by 8.40% and 9.62% on all NLG metrics, respectively, compared to the previous models. The most significant improvement is observed in ROUGE-L, with an increase of 8.27% for R2Gen and 11.83% for M2Tr. This indicates that our method produces more grammatically correct captions, which is particularly important in medical reports where language errors can have serious consequences.
We find that the highest order n-grams (i.e., n=3, 4) show the most significant improvements. This suggests that incorporating external prior information is especially beneficial for generating fluent and informative sentences that typically contain longer phrases and more complex structures.
Overall, our findings demonstrate that integrating external prior information can enhance the performance of existing NLG models for medical image reporting tasks, resulting in more informative and accurate medical reports. By incorporating additional domain-specific knowledge into the models, we are able to generate more precise and informative reports while minimizing computational overhead and training data requirements.

heading: 5 Analysis
text: In this section, we compare the ground truths, synthetic reports created by our proposed model, and two previously published models, R2Gen and M2Tr, to assess the effectiveness of our model in generating concise and accurate reports without irrelevant or false priors.
Table 5 in the Appendix provides examples of synthetic reports generated by each model and the corresponding ground truth for the same radiology image. The first two rows compare reports generated by R2Gen and our model with prior infusion. We observe that R2Gen generates false prior expressions, such as "compared to prior examination", "unchanged from prior", and "again unchanged", which refer to non-existent prior exams. In contrast, our model generates more concise and accurate re-
ports without any prior expressions, resulting in higher performance in NLG metrics.
Similarly, the last two rows of Table 5 in the Appendix compare reports generated by M2Tr and our proposed model. M2Tr produces reports with false prior expressions, such as "present on the previous exam" and "again noted", while our model avoids including any comparison phrases. Furthermore, reports that include prior expressions tend to be longer due to the additional explanations required for comparison. However, the report generation model does not actually have access to previous exams for comparison, rendering the inclusion of prior expressions irrelevant or misleading. As a result, our models can directly control these phrases by conditioning the generation through priors.
Overall, the synthetic reports generated by our proposed model are more concise and accurate compared to those generated by R2Gen and M2Tr, as evidenced by the higher performance in NLG metrics. Our model achieves this by avoiding irrelevant or false prior expressions through a rule-based labeler and generating reports that contain only relevant and accurate information. These succinct and precise reports generated by our model will effectively assist radiologists in their practice.

heading: 6 Limitations and Ethical Considerations
text: Our proposed method has certain limitations and ethical considerations that merit discussion. The effectiveness of our approach heavily relies on the rule-based labeler. However, it is important to acknowledge that the labeler may not capture unseen patterns or variations, potentially limiting improvements in various evaluation metrics. Moreover, we were unable to conduct a comprehensive human evaluation of the rule-based labeler in this study due to resource constraints. Therefore, future work should include a detailed evaluation to assess its performance and address any potential limitations.
Collaboration with three radiologists at Kyoto University is a critical aspect of our work. The regular expressions designed in the rule-based labelers were validated through mutual confirmation by computer scientists and radiologists. However, it is essential to note that the radiologists involved in the collaboration primarily work in a Japanese hospital setting. This may introduce potential biases or patterns that are specific to the local context. Therefore, it is necessary to cross-check the performance of the rule-based labeler with radiologists
from different regions and healthcare systems to ensure broader applicability and minimize any potential bias.
Regarding the datasets used in our study, we exclusively utilized publicly available datasets that are properly anonymized and de-identified, addressing privacy concerns. However, it is crucial to emphasize that if datasets containing comparison exams become available in the future, additional precautions must be taken to ensure that no personally identifiable information is inadvertently disclosed or used in a manner that could identify individual patients.
By acknowledging these limitations and ethical considerations, we aim to encourage future research and discussions in the field, driving advancements in radiology report generation while prioritizing patient privacy, accuracy, and fairness. These considerations will contribute to the development of robust and ethically sound approaches in radiology report generation.

heading: 7 Conclusion
text: In this study, we present a novel approach to generate medical reports from chest X-ray images, aiming to bridge the gap between radiologists’ knowledge and the lack of prior information in generation models. To achieve this, we developed a rule-based labeler capable of extracting comparison priors from radiology reports in the IU X-ray and MIMIC-CXR datasets. These priors were subsequently integrated into state-of-the-art models for conditional report generation, allowing our approach to emulate the realistic diagnostic process of radiologists who possess prior information about patients.
Our experimental results demonstrate the superiority of our method over previous state-of-the-art models, as indicated by improved performance in terms of NLG metrics and a significant reduction in the occurrence of falsely referred prior exams. Through our analysis, we show that the incorporation of comparison priors leads to the generation of more accurate and concise reports, thereby holding great potential to enhance the quality and efficiency of medical report generation for chest X-ray images. Ultimately, this advancement benefits healthcare professionals and patients by providing more reliable and informative reports.
Furthermore, our work highlights the future potential of generating medical reports in an end-to-
end fashion if a dataset containing all previous exams becomes available. The ability to leverage comprehensive prior information would further amplify the accuracy and effectiveness of medical report generation, paving the way for improved healthcare outcomes.

heading: Future Work
text: Furthermore, our work highlights the future potential of generating medical reports in an end-to-
end fashion if a dataset containing all previous exams becomes available. The ability to leverage comprehensive prior information would further amplify the accuracy and effectiveness of medical report generation, paving the way for improved healthcare outcomes