KnowLab at RadSum23: comparing pre-trained language models in radiology report summarization
Year: 2023

Abstract:
This paper presents our contribution to the RadSum23 shared task organized as part of the BioNLP 2023. We compared state-of-theart generative language models in generating high-quality summaries from radiology reports. A two-stage fine-tuning approach was introduced for utilizing knowledge learnt from different datasets. We evaluated the performance of our method using a variety of metrics, including BLEU, ROUGE, Bertscore, CheXbert, and RadGraph. Our results revealed the potentials of different models in summarizing radiology reports and demonstrated the effectiveness of the two-stage fine-tuning approach. We also discussed the limitations and future directions of our work, highlighting the need for better understanding the architecture design’s effect and optimal way of fine-tuning accordingly in automatic clinical summarizations.

text: The 22nd Workshop on Biomedical Natural Language Processing and BioNLP Shared Tasks, pages 535–540 July 13, 2023 ©2023 Association for Computational Linguistics

heading: 1 Introduction
text: Summarization of radiology reports is a useful tool for both doctors and patients. It allows doctors to quickly prioritize and extract essential information from long documents, saving time and improving patient outcomes. This is especially beneficial in medical settings, where time is often limited and there may be a large volume of documents to review.
The rapid development of natural language processing (NLP) techniques, particularly large generative language models such as GPT (Radford et al., 2018), have significantly advanced the field of automatic text summarization. However, there is still much to explore when it comes to generating summaries for specific domains or tasks. For example, generating radiology reports is a challenging task that requires specialized knowledge and language expertise. Radiology reports are typically written in a specific format and include complex medical terminology, which makes it difficult for traditional
summarization techniques to produce accurate and comprehensive summaries.
In the past, several methods have been proposed for radiology report summarization. Chen et al. (2018) developed a method for generating summaries of radiology reports using an attentionbased neural network. Zhang et al. (2018) explored the problem using an augmented pointergenerator model resulted in high overlap with human-generated references. Based on this work, MacAvaney et al. (2019) then built an ontologyaware pointer-generator, which led to improved summarization quality. Furthermore, Delbrouck et al. (2022b) proposed ViLMedic, which is a replicable pipeline that can reproduce the latest results in various medical tasks using multimodal data resources (images and texts) for generating radiology reports.
This paper investigates transfer learning methodologies for adapting large generative language models to the specific domain of radiology report generation. Our main contribution involves a two-stage fine-tuning procedure on a large corpus of radiology reports to handle the specialized language and structure of these reports. We conducted experiments using two clinical datasets: MIMIC-III, which includes various kinds of radiology reports, and MIMIC-CXR, which includes only chest xray radiology reports. The work and results are presented as part of the RadSum23 shared task at BioNLP 2023 (Delbrouck et al., 2023).

heading: 2 Task Description and Dataset
text: The goal of this research is to generate a summary ("Impression" section in Table 1) that highlights the key observations and conclusions of the radiology study. They are generally written by a radiologist after analyzing medical images such as X-rays or CT scans ("Findings" section in Table 1). Our group focused on the textual data and work only on the text-based radiology reports.
535
Two datasets are used in this challenge: the MIMIC-III and MIMIC-CXR Radiology Report Summarization datasets (Johnson et al., 2016, 2019). MIMIC-III radiology reports contains freetext radiology reports from the Beth Israel Deaconess Medical Center. It contains radiology reports from various modality-anatomy including head, abdomen chest, etc. MIMIC-CXR radiology reports, which is a subset of MIMIC-IV dataset, that only contains chest radiology reports.
In terms of the evaluation, the test-sets has been split into two by the organizers. The details of data description can be found in Table 2.

heading: 3 Methods
text: We applied pre-trained language models that are specific for generative tasks. The models are finetuned for the task of radiology report summarization. Specifically, we fine-tuned two different types of pre-trained models - BART (Lewis et al., 2019) and T5 (Raffel et al., 2020). By fine-tuning these models on a large corpus of radiology reports, we
aim to generate high-quality summaries that capture the most important information in the reports.
The BART model, developed by Facebook (Lewis et al., 2019), is a denoising auto-encoder designed for pre-training sequence-to-sequence (Seq2Seq) models. It combines the state-of-theart (SOTA) performance of both the BERT (Devlin et al., 2018) and GPT (Radford et al., 2018) models, inheriting the bidirectional encoder and left-to-right decoder models’ benefits. Because it is constructed on the Seq2Seq Transformers architecture, BART is an excellent choice for abstractive summarization, as it can generate novel text and paraphrase the input text. BART’s original pre-training involved masked language modeling (MLM) with six noises, an enhancement of the BERT model’s single noise masking strategy, making it less likely to learn biased information.
T5, short for Text-To-Text Transfer Transformer, was proposed by Google and utilizes the standard transformer architecture pre-trained on denoising, where spans of text are replaced with a drop token. T5 was trained using a "text-to-text" approach, which means it can perform a wide range of tasks by converting input text into output text, such as language translation, summarization, question answering, and even programming tasks. T5 is also known for its ability to perform well with few-shot learning, meaning it can quickly adapt to new tasks with just a few examples.
BART and T5 are both transformer-based architectures with subtle differences in their layer configurations. T5 is trained as a causal language model (i.e., predicting next tokens), while BART is trained on a masked language modeling objective. Their performances vary depending on the specific tasks.
To optimize the performance and generalizability of our approach, we employed a two-stage fine-tuning process using two different datasets - MIMIC-III and MIMIC-CXR. In the first stage, we fully fine-tuned the pre-trained model on the MIMIC-III dataset using the typical fine-tuning approach. This involved training the model on the dataset to adapt it to the specific task of radiology report summarization. In the second stage, we performed another round of fine-tuning on the MIMICCXR dataset by freezing the last two layers in the encoder and decoder. It is assumed that freezing the last two layers can help prevent the model from overfitting the training data. Lower layers tend to
learn more generalizable feature representations, while higher layers may be more prone to overfitting on the specific patterns of the training data. By freezing the last two layers, we can limit the degree of fitting to the training data, enabling the model to generalize better on new, unseen data.
Overall, the two-stage fine-tuning process is a key component of our approach, allowing us to leverage the strengths of both datasets and optimize the performance of the model on the task of radiology report summarization.

heading: 3.1 Experiment Setup
text: For comparison, we fine-tuned BART base model1 and BioBART model 2(Yuan et al., 2022), T5 base model3 and SciFive model 4(Phan et al., 2021). BioBART is a generative language model that adapts BART to the biomedical domain by pretraining on large PubMed corpora. SciFive is a domain-specific T5 model that pretrained on large biomedical corpora.
We tuned the hyper-parameters during the training phase to optimize the performance of our approach. The initial learning rate is 2e-5. The maximum epochs used for training is set to 20 with batch size of 16. The maximum length for input data is 1024. The maximum length for output is 128. The beam size is set to 5, and no_repeat_ngram_size is set to 2. Other hyper-parameters are set as their default values.

heading: 3.2 Evaluation Metrics
text: According to the instructions, we consider five evaluation metrics for this work, including BLEU, Rouge, BERT score, CheXbert, and RadGraph (Papineni et al., 2002; Lin, 2004; Zhang et al., 2019; Smit et al., 2020; Delbrouck et al., 2022a).
BLEU and Rouge scores measures the overlap between the generated summary and references based on n-grams (Papineni et al., 2002; Lin, 2004). The main difference between Rouge and BLEU is that Rouge emphasizes recall and BLEU focuses on precision. This means that Rouge is more focused on capturing the important content of the summary, while BLEU is more focused on ensuring that the summary is grammatically correct and fluent. BERT score measures the similarity between the embeddings of the machine-generated text and
1https://huggingface.co/facebook/bart-base 2https://huggingface.co/GanjinZero/biobart-base 3https://huggingface.co/t5-base 4https://huggingface.co/razent/SciFive-base-Pubmed
the references based on the contextual embeddings generated by the BERT model (Zhang et al., 2019). CheXbert revised the BERT score by adding expert annotations on Chest X-rays (Smit et al., 2020). RadGraph is a metric used for evaluating radiology report generation (Delbrouck et al., 2022a). It provides better domain-adjusted evaluation based on a novel Information Retrieval(IE) method (i.e. entities and relationships) from MIMIC-CXR dataset.

heading: 4 Experiments and Results
text: Table 3 and Table 4 present the experimental results obtained from the MIMIC-III and MIMIC-CXR datasets. In the MIMIC-III experiments, we performed fine-tuning on four models and evaluated their performance on the two test sets: MIMIC-III test set and the MIMIC-III hidden test set. The BART model achieved the highest overall performance on both test sets, while the BioBART model achieved the highest Bertscore on the MIMIC-III test set and the highest RadGraph score on the MIMIC-III hidden test set.
For the MIMIC-CXR experiments, we performed a second round of fine-tuning on the models that were previously fine-tuned on MIMIC-III. The baseline model in Table 4 refers to the model that underwent only one round of fine-tuning using the MIMIC-CXR data. To evaluate the performance, we included the CheXbert F1 score as it is commonly used for evaluating chest X-ray radiology reports. Overall, the T5 model achieved the best performance on the MIMIC-CXR test set, while the BART and BART_freeze models had higher RougeL and Bertscore. In terms of the MIMIC-CXR hidden test set, T5 achieved the best performance, while T5_freeze had the highest F1 CheXbert score. The results showed that our twostage fine-tuning approach yielded better performance than the baseline model in both test sets.
In summary, our experimental results demonstrated that BART and T5 models have different strengths and may perform better on different datasets or evaluation metrics. This may due to various factors, such as the size and complexity of the dataset, the quality of the training data, etc.
There is considerable potential for future improvements to our approach. Firstly, our results suggest that freezing the last two layers of the models may yield some advantages, but this finding requires further investigation to fully understand the effects of this approach. Secondly, we acknowl-
edge that there are limitations to our current training process, and we believe that improvements can be made to further optimize the performance of our models. Specifically, we plan to explore alternative training strategies and hyper-parameter tuning methods to improve the performance of our models on the MIMIC-III and MIMIC-CXR datasets. Additionally, we recognize the importance of testing our models on larger datasets and expanding the scope of our evaluations to include other natural language processing tasks. Finally, we will continue to monitor developments in the field and adapt our methods accordingly to ensure that we remain at the forefront of natural language processing research.

heading: 5 Conclusion
text: This paper presented our investigation into transfer learning for the radiology report summarization task. Specifically, we conducted a comprehensive evaluation of different pre-trained language models which were built by deploying encoder-decoder architectures. We introduced a two-stage fine tuning
methodology which involved BART, T5, and their variants in the biomedical domain, and compared their performances on the summarization task.
We also evaluated the effectiveness of our approach using various metrics, such as BLEU, ROUGE, Bertscore, CheXbert, and RadGraph, and demonstrated that our method achieved high performance on the summarization task. In fact, our approach was ranked third on the leaderboard of the RadSum23 MIMIC-CXR hidden test set, highlighting the competitiveness of our approach compared to other methods.

heading: Acknowledgements
text: We would like to thank the BioNLP 2023 RadSum23 Shared Task organizers who provide the valuable resources including the baseline pipeline ViLMedic (Delbrouck et al., 2022b) and this exciting tasks (Delbrouck et al., 2023).

heading: Future Work
text: There is considerable potential for future improvements to our approach. Firstly, our results suggest that freezing the last two layers of the models may yield some advantages, but this finding requires further investigation to fully understand the effects of this approach. Secondly, we acknowl-
edge that there are limitations to our current training process, and we believe that improvements can be made to further optimize the performance of our models. Specifically, we plan to explore alternative training strategies and hyper-parameter tuning methods to improve the performance of our models on the MIMIC-III and MIMIC-CXR datasets. Additionally, we recognize the importance of testing our models on larger datasets and expanding the scope of our evaluations to include other natural language processing tasks. Finally, we will continue to monitor developments in the field and adapt our methods accordingly to ensure that we remain at the forefront of natural language processing research