shs-nlp at RadSum23: Domain-Adaptive Pre-training of Instruction-tuned LLMs for Radiology Report Impression Generation
Year: 2023

Abstract:
Instruction-tuned generative large language models (LLMs), such as ChatGPT and Bloomz, possess excellent generalization abilities. However, they face limitations in understanding radiology reports, particularly when generating the IMPRESSIONS section from the FINDINGS section. These models tend to produce either verbose or incomplete IMPRESSIONS, mainly due to insufficient exposure to medical text data during training. We present a system that leverages large-scale medical text data for domain-adaptive pre-training of instructiontuned LLMs, enhancing their medical knowledge and performance on specific medical tasks. We demonstrate that this system performs better in a zero-shot setting compared to several pretrain-and-finetune adaptation methods on the IMPRESSIONS generation task. Furthermore, it ranks 1st among participating systems in Task 1B: Radiology Report Summarization at the BioNLP 2023 workshop.

text: The 22nd Workshop on Biomedical Natural Language Processing and BioNLP Shared Tasks, pages 550–556 July 13, 2023 ©2023 Association for Computational Linguistics
Instruction-tuned generative large language models (LLMs), such as ChatGPT and Bloomz, possess excellent generalization abilities. However, they face limitations in understanding radiology reports, particularly when generating the IMPRESSIONS section from the FINDINGS section. These models tend to produce either verbose or incomplete IMPRESSIONS, mainly due to insufficient exposure to medical text data during training. We present a system that leverages large-scale medical text data for domain-adaptive pre-training of instructiontuned LLMs, enhancing their medical knowledge and performance on specific medical tasks. We demonstrate that this system performs better in a zero-shot setting compared to several pretrain-and-finetune adaptation methods on the IMPRESSIONS generation task. Furthermore, it ranks 1st among participating systems in Task 1B: Radiology Report Summarization at the BioNLP 2023 workshop.

heading: 1 Introduction
text: A radiology report is the primary method for radiologists to communicate medical image interpretations (e.g., X-rays) and conclusions to ordering physicians. These reports typically include several sections (Kahn Jr et al., 2009), with the most important ones being the FINDINGS and IMPRESSIONS sections. The FINDINGS section describes abnormalities and diagnoses, while the IMPRESSIONS section summarizes the findings and highlights major abnormalities and recommendations. A sample report with these sections can be found in Table 1.
The field of radiology has experienced rapid growth in NLP techniques due to the need for efficient and accurate analysis of radiological reports, which often consist of unstructured textual data (Ghosh et al., 2023). For instance, various efforts have been made to automatically generate
∗ Equal Contribution.
IMPRESSIONS from FINDINGS, such as those by Zhang et al. (2018) using neural seq2seq, Karn et al. (2022) employing multi-agent reinforcement learning, and Delbrouck et al. (2023) utilizing Pretrained language models (PLMs) with BERT-based systems.
PLMs are trained on vast, diverse corpora, allowing them to capture various linguistic patterns (Gururangan et al., 2020). Despite their strengths, they have limitations and biases affecting out-of-domain tasks. Domain-specific PLMs, like BioBERT (Lee et al., 2020) and RadBERT (Yan et al., 2022), have been developed for biomedical and clinical NLP. However, these models face constraints, such as RadBERT’s training on small datasets with limited anatomical specialties.

heading: The pretrain-and-finetune paradigm for PLMs
text: has emerged as the prevailing approach to address downstream tasks with substantial training data scarcity (Karn et al., 2021). Recent studies (e.g., Gururangan et al. (2020)) suggest that additional pretraining on in-domain text, known as specialist pretraining, is more effective in enhancing downstream performance. Currently, another approach
550
has surfaced, where instead of finetuning PLMs to perform downstream tasks, the objectives of downstream tasks are reformulated using textual prompts similar to the original pre-training objectives (Liu et al., 2023). This pretrain-and-prompttune paradigm is commonly referred to as prompttuning.
Multitask prompted finetuning (also known as instruction tuning) is a type of large-scale pretrainand-prompt-tune paradigm in which large PLMs (also referred to as LLMs) are finetuned using datasets representing various NLP tasks, defined by instructions as natural language prompts (Scao et al., 2022). Utilizing this approach, Scao et al. (2022) endowed their LLM, Bloom, with the ability to perform multilingual zero-shot instruction-based tasks and labeled it Bloomz.
We propose an extension to the domain adaptation paradigm beyond the typical method of pretrain-and-finetune for domain-specific tasks. We posit that further pretraining of instructiontuned LLMs, already subjected to the pretrainand-prompt-tune process, with in-domain text will improve and simplify adaptation. We refer to our approach as general-pretrain-prompt-tune-andspecial-pretrain. In this approach, the model is trained using the same initial LM objective in each of the three training stages (i.e., general pretraining, prompt-tuning, and domain-specialized pretraining), which is a significant advantage.
We continued pretraining the instruction-tuned Bloomz on MIMIC-IV to form RadBloomz and evaluated this adaptation paradigm on the radiology report summarization task. The proposed system, in a zero-shot setting, demonstrates better performance than pretrain-and-finetune methods and ranks 1st among participating systems in Task 1B: Radiology Report Summarization at the BioNLP 2023 workshop.
Overall, our contributions are as follows:
• We expand the domain adaptation paradigm by introducing general-pretrain-prompt-tuneand-special-pretrain, which further pretrains instruction-tuned LLMs, such as Bloomz, on domain-specific text.
• We demonstrate that the new adaptation paradigm for an instruction-tuned LLM in radiology yields better performance in a zeroshot setting compared to the pretrain-andfinetune methods.

heading: 2 Datasets

heading: 2.1 Pretraining Datasets for Radiology Domain Adaptation.
text: We performed domain-adaptive pretraining using the recently published MIMIC-IV radiology reports dataset (Johnson et al.; PhysioBank, 2000). This dataset contains over 2.3 million radiology reports from 237k patients and amounts to approximately 616 million tokens using the Bloomz tokenizer (Muennighoff et al., 2022). After preprocessing, we utilized only 1.4 million reports with 190 million tokens. Further details on the dataset statistics can be found in Table 2.

heading: 2.2 Finetuning Datasets for Impression Generation
text: We utilized the datasets shared for Task 1B: Radiology Report Summarization at the BioNLP 2023 workshop for our fine-tuning task. This task comprises three datasets: MIMIC-III (Johnson et al., 2016), MIMIC-CXR (Johnson et al., 2019), and CHEXPERT (Irvin et al., 2019), pre-split into FINDINGS and IMPRESSIONS sections. For MIMICIII, there are 59,320 reports in the training dataset, 7,413 in validation, 6,526 in the test set, and 6,531 in the hidden test set. Most reports (91.4%) pertain to CT imaging, with the most represented anatomy being the head (52.8%). Although the task related to MIMIC-CXR/CHEXPERT datasets is multimodal, we only used radiology reports for fine-tuning and inference. The MIMIC-CXR training dataset has 125,417 reports for training, 991 for validation, and 1,624 for testing. The hidden test dataset, a CHEXPERT dataset, contains 1,000 reports for evaluation.

heading: 3 Methods
text: Our methods comprise preprocessing, domainadaptive pretraining, fine-tuning, and inference.

heading: 3.1 Preprocessing
text: In the preprocessing step, Regex-based cleaning and normalization were used to remove irrelevant
characters and texts from the report. Special tokens for de-identified text were incorporated, and distinct sections, such as FINDINGS and IMPRESSIONS, were identified. Reports with containing both sections and fewer than 512 tokens were selected.

heading: 3.2 Domain adaptive pretraining (DAPT)
text: We chose GPT-powered Bloom (Scao et al., 2022) as our study’s base LLM. Bloom, a multilingual LLM adapted from Megatron-LM GPT2 (Shoeybi et al., 2019), has multiple versions based on parameters. The largest has 176 billion parameters, 70 layers, 112 attention heads, and 14,336- dimensional hidden layers. We use Bloomz (Muennighoff et al., 2022), a massive multitask instruction-tuned version of Bloom, specifically its Bloomz-7b1 variant with 7 billion parameters, 30 layers, and 4,096-dimensional hidden layers for domain adaptation.
Following our proposed pretrain-fine-tune-andpretrain paradigm, we continuously pretrain Bloomz-7b1 using cross-entropy loss on autoregressively generated tokens from the FINDINGS and IMPRESSIONS sections of MIMIC-IV reports.

heading: 3.3 Finetuning
text: In this study, the domain-specific task for finetuning an LLM is Radiology Report Summarization. Using standard prompt-based fine-tuning, we employ FINDINGS and TL;DR as prompt, and finetune Bloomz-7b1 by comparing auto-regressively generated summary tokens to ground-truth IMPRESSIONS using cross-entropy loss. This method ensures fine-tuning consistency with base Bloom and intermediate Bloomz’s pretraining and instructiontuning objectives. To prevent catastrophic forgetting, Bloomz-7b1’s trainable parameters are minimized by only allowing the last layer to be modified.

heading: 3.4 Inference
text: The inference pipeline utilizes the trained model to generate IMPRESSIONS based on the given FINDINGS. Evaluation metrics for the generated results include Rouge scores (Lin, 2004), F1RadGraph (Delbrouck et al., 2022a), Bertscore (Zhang et al., 2019), and F1CheXbert (Xie et al., 2023) for the MIMIC-CXR and CHEXPERT datasets.

heading: 4 Experiments
text: We propose two experimental runs for the summarization task.
1. Radiology Domain Adaptive Pretraining (RadBloomz) with MIMIC-IV and zeroshot inference. The Bloomz-7b1 model is fine-tuned with a causal language objective on MIMIC-IV radiology reports, creating RadBloomz. With a sequence length of 512, training batch size of 64, validation batch size of 32, learning rate of 3e-5, and AdamW optimizer (Loshchilov and Hutter, 2017), the best zero-shot inference is achieved at 24k steps.
2. RadBloomz finetuned with MIMIC-III. Following the pretrain-and-finetune paradigm, RadBloomz is further fine-tuned with the MIMIC-III dataset for radiology report summarization. Using the same hyperparameters and training configuration, the best results are achieved at 2697 steps.
All experiments were conducted on the same infrastructure, utilizing eight Tesla A100 SXM4 GPUs (80GB memory each) and Deepspeed zero-3 configuration (Rasley et al., 2020) with BF16 enabled. A sampling-based technique was used to generate summaries from the model output distribution, with a maximum of 128 tokens, top_k set to 50, and top_k at 0.7.

heading: 5 Results and Discussion
text: We evaluated RadBloomz against other systems using ROUGE for n-gram overlap and F1RadGraph for fact overlap.
Table 3 showcases RadBloomz’s (shs-nlp team) performance on MIMIC-CXR and MIMIC-III hidden test datasets. The MIMIC-III hidden dataset contains only reports, while MIMIC-CXR includes reports and images. Since our system is text-based, MIMIC-III is a more suitable evaluation. RadBloomz ranks first in MIMIC-III and fourth in MIMIC-CXR among all submitted systems, demonstrating the effectiveness of our domain adaptation technique in a multi-modal task.
In Table 4, we compare the performance of the standard pretrain-and-finetune approach with our proposed pretrain-prompt-tune-pretrain-and-zeroshot paradigm on the radiology report summariza-
tion test data for Task 1B challenge.1 We observe that while finetuning with MIMIC-III improves Rouge-L and Bertscore metrics for the MIMICIII test dataset, Bertscore, F1-RadGraph, and F1cheXbert scores are lower for the finetuned model. This indicates that the domain adaptation paradigm is sufficient for achieving higher performance without requiring task-specific finetuning.

heading: 5.1 Error Analysis
text: A thorough error analysis on the open test datasets reveals that many generated IMPRESSIONS receive low scores for both Rouge and F1-RadGraph when the ground-truth radiology report IMPRESSIONS does not mention any abnormalities. For instance, the generated impression “normal MRI of the cervical spine” and the ground truth impression “negative study” are semantically similar. However, these n-gram overlap-based scores fail to recognize their semantic relatedness.
Similarly, we observed that similar FINDINGS sometimes generate different impressions. For example, impressions can be as detailed as: “near complete opacification of the ethmoid air cells and sphenoid sinuses, moderate air-fluid level with mucosal thickening of the right maxillary sinus, and moderate mucosal thickening of the left maxillary sinus.” Meanwhile, similar findings in another re-
1Ground-truths are available only for the open test data for any additional evaluation.
port might be summarized as “pansinusitis, as described above.”
The detailed error analysis has also uncovered several intriguing types of hallucinations by RadBloomz on theMIMIC-III test dataset. More details can be found in the Appendix, Section A.1.

heading: 6 Conclusion
text: In this work, we introduce a new domain adaptation paradigm of general-pretrain-prompt-tuneand-special-pretrain, where we further pretrain an instruction-tuned LLM (Bloomz) on radiology domain text. We use radiology report summarization as the domain-specific task and demonstrate that the new paradigm-based LLM model outperforms the standard "pretrain-and-finetune" method, even in a zero-shot setting. The system ranks 1st among participating systems in the hidden-test category for Task 1B: Radiology Report Summarization at the BioNLP 2023 workshop.

heading: Limitations
text: There are a few limitations pertaining to the training data we used, some of which are listed below.
1. Our domain adaptation of LLMs was performed on English reports only; therefore, it may not work out of the box in a multilingual setting.
2. The paper utilizes the MIMIC-IV dataset for DAPT training, which might include overlapping data from MIMIC-III and MIMICCXR. Consequently, there is a potential risk of information leak in this method.
3. There is a data imbalance concerning imaging modalities and anatomies covered by our training data. For example, regions such as extremities, neck, spine, and shoulder are underrepresented in the dataset, and report summarization related to those regions needs thorough evaluation.
4. A study is needed to examine the diversity of patients represented in the data and how it impacts the model’s performance for underrepresented communities.
5. Different radiologists and radiology departments have distinct preferences and styles for writing reports. Moreover, clinical referrals occasionally dictate the extent to which certain details are documented in the report. No study has been conducted on the consistency, uncertainty, or information richness of the report.
Aside from the training data, the model’s space and time throughputs may render them unsuitable for on-premise and/or at-the-edge applications. This aspect presents an opportunity for further research on how to best quantize and deploy RadBloomz (and similar LLMs) within the clinical workflow to enhance efficiency for radiologists. Additionally, the paper utilizes the MIMIC-IV dataset for DAPT training, which could contain overlapping data from MIMIC-III and MIMICCXR. Consequently, there is a potential risk of information leak in this method.

heading: Ethics Statement
text: The research performed in this paper adheres to the Association for Computing Machinery (ACM) Code of Ethical and Professional Conduct 2 adopted by the Association for Computational Linguistics (ACL). To prevent any harm caused due to errors in our model-generated outputs, our models are meant to be deployed in a human-in-theloop setting where the key information extracted by our models are reviewed by radiologists and physicians.
2https://www.acm.org/code-of-ethics

heading: Disclaimer
text: The concepts and information presented in this paper/presentation are based on research results that are not commercially available. Future commercial availability cannot be guaranteed.

heading: Acknowledgements
text: We sincerely thank our DTI team for their support, the Siemens Healthineers supercomputing team for the training infrastructure, and anonymous reviewers for valuable feedback. We also acknowledge the shared task organizers and ViLMedic (Delbrouck et al., 2022b) for their vital contributions and efforts in ensuring a smooth, productive experience for all shared task participants.

heading: A Appendix

heading: A.1 Error Analysis
text: A detailed error analysis on the MIMIC-III test dataset has uncovered several intriguing types of hallucinations generated by the RadBloomz model. These hallucinations can be categorized as follows:

heading: A.1.1 Numerical Hallucination
text: The RadBloomz model has been observed to hallucinate, particularly when numbers are involved. It struggles with comparing numbers or associating mentioned numbers with their correct concepts. One example can be seen in Table 5, where the
model has difficulty understanding the largest size of the lymph node conglomerate mentioned in the text, even though it is explicitly stated which one is the largest conglomerate.

heading: FINDINGS

heading: A.1.2 Missing Context and Facts
text: There are FINDINGS in which the RadBloomz model has overlooked facts or failed to understand the context of a sentence well. One such example is illustrated in Table 6.

heading: A.1.3 Hallucination of New Facts
text: New concepts, unrelated to what is mentioned in the findings, are a part of the summary. These
occurrences are rare. We found only 6 cases out of the 6.5k where this occurred. One such example is shown in Table 7.

heading: FINDINGS

heading: Future Work
text: Future commercial availability cannot be guaranteed