ELiRF-VRAIN at BioNLP Task 1B: Radiology Report Summarization
Year: 2023

Abstract:
This paper presents our system at the Radiology Report Summarization Shared Task-1B of the 22nd BioNLP Workshop 2023. Inspired by the work of the BioBART model, we continuously pre-trained a general domain BART model with biomedical data to adapt it to this specific domain. In the pre-training phase, several pre-training tasks are aggregated to inject linguistic knowledge and increase the abstractivity of the generated summaries. We present the results of our models, and also, we have carried out an additional study on the lengths of the generated summaries, which has provided us with interesting information.

text: The 22nd Workshop on Biomedical Natural Language Processing and BioNLP Shared Tasks, pages 524–529 July 13, 2023 ©2023 Association for Computational Linguistics

heading: 1 Introduction
text: Radiology reports are documents that interpret radiological examinations. Usually, a radiology report consists of three sections: (1) a background section that describes general information about the patient and exam, (2) a findings section that presents details of the examination, and (3) an impression section that summarizes the findings against the background. This last section is the most crucial for doctors to make clinical decisions.
Due to the recent success of self-supervised learning, the focus of text summarization research has exhibited a gradual shift from extractive techniques to abstractive techniques. The bestperforming abstractive models are BART (Lewis et al., 2020), T5 (Raffel et al., 2020), PEGASUS (Zhang et al., 2020a), and GPT-3 (Brown et al., 2020), being all of them Transformers (Vaswani et al., 2017) pre-trained self-supervisedly as denoising sequence to sequence autoencoders. This kind of approaches allow to pre-train deep architectures to learn vast amounts of general linguistic knowledge from large corpora, that can be transferred to downstream tasks by means of fine-tuning. Almost all of these systems used benchmark datasets compiled from news articles, such as the CNN-
DailyMail dataset (CNN-DM) (Hermann et al., 2015) and NEWSROOM (Grusky et al., 2018). However, not so many efforts have been carried out in the biomedical domain.
Language models pre-trained on biomedical corpora may further enhance the performance of current biomedical NLG methods, such BioBERT (Lee et al., 2020) or PubMedBERT (Gu et al., 2021). However, there are very few in-domain generative language models for biomedicine. In (Yuan et al., 2022), authors proposed a biomedical autoregressive generative language model, BioBART, pre-trained on the biomedical corpora. They continuously pre-train BART on PubMed1 abstracts to achieve biomedical domain adaption only using the text-infilling task. The in-domain BioBART outperforms BART model and sets strong baselines for several NLG tasks.
In the framework of BioNLP workshop, some challenges and shared tasks focusing on summarization were created. MEDIQA 2019 edition focused on question entailment and textual inference and their applications in medical Question Answering (Ben Abacha et al., 2019). MEDIQA 2021 (Ben Abacha et al., 2021) promoted research on summarization for consumer health QA and clinical text. In this edition, the winner system (Dai et al., 2021), based on PEGASUS, employed a domain adaptation strategy by further fine-tuning a small amount of in-domain data to improve generalization and transfer abilities.

heading: 2 Task Description
text: The Shared Task-1B of the 22nd BioNLP Workshop 2023 (Delbrouck et al., 2023), focuses on the summarization of radiology reports. The task of the summarization of radiology reports can be defined as follows: given a radiology report with findings and background sections, the goal is to
1https://pubmed.ncbi.nlm.nih.gov/
524
generate the impression section. For this shared task, the Impressions are only generated from the Findings section.
Shared Task 1B was divided into two subtasks. The first one is about generating impressions sections based exclusively on the text report. The second one is summarizing the report based on the text information and the indicators that could be extracted from the attached radiology image. The participants were invited to approach both subtasks but were allowed to participate in one; we chose to participate only in the first subtask.

heading: 2.1 The Dataset
text: For the subtask where we participated, a dataset is provided based on MIMIC-III (Johnson et al., 2016) with 79 779 samples from two different radiography modalities and six anatomical parts. This dataset was split into four partitions: train (59 320 samples), validation (7413), test (6526), and hidden-test (6531).
Table 1 details the average number of sentences and words for Findings and Impressions, excluding the Impressions of the hidden-test that were not available to participants. On the one hand, we notice that train and validation have similar lengths for both, Impressions and Findings. On the other hand, the test partitions contain longer Findings, especially the hidden-test. Moreover, Impressions are shorter in the test partition than in the train and validation ones; thus, test presents a higher compression ratio in its Impressions partition than those of the other two partitions.

heading: 2.2 System Evaluation
text: For evaluating the systems, the ViLMedic framework (Delbrouck et al., 2022b) was used. It is a framework that aims to increase results reproducibility for medical tasks, such as medical
report summarization. Specifically, the systems were evaluated with the following metrics and scores: ROUGE-L (Lin, 2004), BLEU (Papineni et al., 2002), BERTScore (Zhang et al., 2020b), and RadGraph (Delbrouck et al., 2022a).

heading: 3 Pre-training Model
text: Inspired by the work of the BioBART model, we continuously pre-trained a general domain BART model with biomedical data to adapt it to this specific domain. Specifically, our starting point was the architecture and weights of the base version of BART2, publicly available at the repository of HuggingFace (Wolf et al., 2020).
For the pre-training phase, we followed the methodology used in the News Abstractive Summarization models (NAS) work (Ahuir et al., 2021). In NAS, several pre-training tasks are aggregated to inject linguistic knowledge during the pre-training stage and to increase the abstractivity of the generated summaries. We chose this methodology because we hypothesize that reference impressions are written in a mostly abstractive way. Also, the pre-training method helped to transfer more knowledge to the summarization task, which increased the model’s performance in the original work.
For pre-training, we chose data that were as similar as possible to radiology reports in terms of vocabulary and grammar. We selected the following MIMIC datasets available at PhysioNet (Goldberger et al., 2000): note events in MIMICIII (2 083 180 samples) (Johnson et al., 2016), radiology reports in MIMIC-CXR (128 032) (Johnson et al., 2019), and discharge (331 794) and radiology reports (2 321 355) in MIMIC-IV (Johnson et al., 2023). Additionally, we included Wikipedia articles related to medicine to reinforce the domain vocabulary (97 192).
The base version of the BART model is limited to 1024 input tokens; however, most samples exceeded this size. This fact could lead us to lose valuable training data. To overcome this limitation, we split texts into narrower samples as follows: having the text split by lines and a window of no more than 1000 words, we generated sub-samples
2https://huggingface.co/facebook/bart-base
that contained at least a new line and filled the window with as many words as possible. With this method, we obtained a dataset of 40 894 042 samples.
Due to infrastructure limitations, we could only train the model for one epoch, which took 12 days with four NVIDIA RTX 3090 graphic cards. The following hyperparameters were used: 4 samples per device, 256 gradient accumulation steps, a learning rate of 5×10−5 with a linear scheduler, 1% of the epoch for warm-up, and 32 bits of training precision. For the hyperparameters not mentioned, we have used the default values of the 4.23.1 version of transformers library of HuggingFace3.

heading: 4 Models for the Task
text: For the downstream task, we obtained three models based on our pre-trained model. The first model (M1) was fine-tuned with the train partition of the shared task. The second one (M2) was fine-tuned with the train and validation partitions. Finally, the third one (M3) was fine-tuned with all the partitions with available references: train, validation, and test.
For the fine-tuning phase, we did a grid search of certain hyperparameters with RayTune (Liaw et al., 2018) through the HuggingFace library. We did 20 trials over the following hyperparameters: learning_rate (from 8× 10−6 to 4 × 10−5), num_train_epochs (10 or 15), and gradient_accumulation_steps (2, 4, or 8). Since we wanted to find which models obtained a more balanced performance among the four metrics of the task (ROUGE-L, BLEU, RadGraph-F1, and BertScore-F1), we defined the objective to maximize as the harmonic mean (Ferger, 1931) of these four scores. Finally, the three models were fine-tuned during 15 epochs with an NVIDIA RTX 4090 with the following hyperparameters: 8 samples per device, 4 gradient accumulation steps, and a learning rate of 2.14× 10−5.
For the generation of impressions, we used the generate method4 of HuggingFace. To achieve
3https://huggingface.co/docs/transformers/ v4.23.1/en/main_classes/trainer#transformers. TrainingArguments
4https://huggingface.co/docs/transformers/ v4.23.1/en/main_classes/text_generation# transformers.generation_utils.GenerationMixin
better performance, we identified certain hyperparameters and performed grid search using the M1 model and the validation partition, specifically: max_length (60, 70, 80, 90, or 100), num_beams (3, 4, 6, 8, or 10), and no_repeat_ngram_size (3, 4, 5, 6, 8, or 10). The bolded values maximized the harmonic mean score; thus, we fixed them to generate impressions with ours models.

heading: 5 Results
text: Table 2 shows the results of our models (M1, M2, M3) and those of the groups (G1, G2) that reached the highest score, excluding our models, on any of the four scores: BLEU (BL), ROUGE-L (RL), BertScore-F1 (BS), and RadGraph-F1 (RG). The overall performance on the four metrics is reflected by the harmonic mean (HM). The results are divided in two sections: test (T) and hidden-test (HT). The leaderboard scores were computed by limiting the prediction and the reference to 256 words.
Overall, our models have lower performance than the best systems. In the test partition, our best model (M1) averages a 9% lower performance than the other two groups if BLEU is excluded from the count, and 5.3% less when is included, meaning that M1 performs substantially better in BLEU than the other systems. Comparing our models, M2 seems to perform worse than M1, despite being trained with more data. In the case of hidden-test, our best model (M3) averages 10.7% lower performance than G1 if BLEU is excluded from the count and 8.43% if not. Comparing the performance of our models, unlike what happened in the test partition, M3 performs better than M1. Therefore, the inclusion of the test partition in training resulted in more acknowledgment
for the model, probably because of the additional findings types.

heading: 6 Discussion
text: When we observe Table 2, two main questions rapidly come to our mind: Why did our models obtain lower values in all scores but BLEU?, and Why the M2 model performed worse than M1, despite being trained with more data?. Surprisingly, both questions point to a main problem in our models: the length of the generated impressions.
Figure 1 presents the distribution of the impressions by their length in words. The blue distribution is the reference impressions, the red one is for those generated by M1, and the green one is for M2. It is noticeable that the impressions of our models follow a completely different distribution than the references. However, the three distributions have similar averages in word count: 51 for the references, 49 for the M1, and 51 for the M2. Our models are trying to set a common length for the impressions instead of identifying which ones should be shorter and which ones should be longer. Therefore, BLEU seems to be weaker in this situation than the other metrics. Moreover, M2 generates longer impressions than M1, lowering the precision and, by extension, its general performance. However, there is a chance that M2 excels in some interesting aspects compared to M1.
Table 3 shows the precision and recall obtained by the models M1 and M2 on test for ROUGEL, BERTScore, and RadGraph; also, the harmonic means of these six measures are shown. The SLN group shows the real performance of the models. Contrary, SLY shows the performance when, at most, the first n sentences of the prediction are
taken into account, where n is the number of sentences of the reference. On SLN, we observe that M2 has better recall than M1 but worse precision due to the longer generated impressions, which caused the final lower performance. On SLY, it is noticeable that both models gain more precision than lose recall; thus, our models place more relevant information at the beginning. SLY shows higher harmonic mean values than SLN, which indicates that we could improve the performance of our models by just focusing on making the models increase their focus on the reference length. Moreover, the harmonic mean values also show that M2 places more relevant content than M1 at the beginning of the text because limiting the number of sentences was more beneficial for M2 than for M1. Therefore, the additional data boosted the model in aspects that were not noticeable by using F1 measures.

heading: 7 Conclusions
text: We presented an approach for Radiology Report Summarization that continuously pre-trains a general domain BART model. This approach focuses on two main aspects: the use of biomedical data to adapt the model to this specific domain and the use of several pre-training tasks designed to inject linguistic knowledge and increase the abstractivity of the generated summaries. After the pre-training phase, we fine-tuned this model with different amounts of data from the shared task. We also presented a study of the relationship between the models performance and the lengths of the generated summaries. We observed that our models condense the main information in the first sentences of the summaries. From the length distribution of the summaries, we found that our models tend to generate summaries with a common length; meanwhile, the reference summaries present more length variability. It seems that this behavior could penalize the performance of our
models, especially on those reports with short reference summaries.
Limitations
The pre-training methodology used in this work applies a masking process at the sentence level that requires scoring the relevance of each sentence within the text. Therefore, this implies additional computational costs, limiting the scalability of our approach.
Due to time restrictions, the appearance of hallucinations in the generated radiology reports by our models has not been measured. It would be necessary to quantify this aspect because of the criticality of the domain of use in future works.
Ethics Statement
The additional data that we have used for the pre-training process are from the MIMIC dataset, which meets the ethical requirements of Patient Health Information.

heading: Acknowledgements
text: This work is partially supported by MCIN/AEI/10.13039/501100011033, by the "European Union and “NextGenerationEU/MRR”, and by “ERDF A way of making Europe” under grants PDC2021-120846-C44 and PID2021126061OB-C41. It is also partially supported by the Generalitat Valenciana under project CIPROM/2021/023, and by the Spanish Ministerio de Universidades under the grant FPU21/05288 for university teacher training.

heading: Future Work
text: It would be necessary to quantify this aspect because of the criticality of the domain of use in future works