RadAdapt: Radiology Report Summarization via Lightweight Domain Adaptation of Large Language Models
Year: 2023

Abstract:
We systematically investigate lightweight strategies to adapt large language models (LLMs) for the task of radiology report summarization (RRS). Specifically, we focus on domain adaptation via pretraining (on natural language, biomedical text, or clinical text) and via discrete prompting or parameter-efficient fine-tuning. Our results consistently achieve best performance by maximally adapting to the task via pretraining on clinical text and finetuning on RRS examples. Importantly, this method fine-tunes a mere 0.32% of parameters throughout the model, in contrast to end-to-end fine-tuning (100% of parameters). Additionally, we study the effect of in-context examples and out-of-distribution (OOD) training before concluding with a radiologist reader study and qualitative analysis. Our findings highlight the importance of domain adaptation in RRS and provide valuable insights toward developing effective natural language processing solutions for clinical tasks.

text: The 22nd Workshop on Biomedical Natural Language Processing and BioNLP Shared Tasks, pages 449–460 July 13, 2023 ©2023 Association for Computational Linguistics

heading: 1 Introduction
text: Radiology reports are comprehensive documents that capture and interpret the results of a radiological imaging examination. Reports are often structured into three main sections: (1) a background section that provides general information about the exam and the patient, e.g. medical history (2) a findings section that presents detailed exam analysis and results, and (3) an impression section that concisely summarizes the most salient findings. In a typical workflow, a radiologist first dictates the detailed findings and then distills them into a concise impression. This impression is the most significant part of a radiology report, as it contains crucial information for clinical decisionmaking (Kahn Jr et al., 2009). However, perform-
*Equal contribution
ing radiology report summarization (RRS) manually can be labor-intensive and prone to errors (Gershanik et al., 2011), motivating the importance of automating this task.
Large language models (LLMs) have demonstrated remarkable capabilities in natural language understanding and generation, serving as foundation models that can be adapted to various domains and tasks. However, their sheer size, sometimes exceeding 100B parameters, makes training for domain-specific tasks prohibitively expensive in terms of computation and training data. We address this challenge by exploring lightweight strategies for domain adaptation in the context of RRS, culminating in the following contributions:
• We systematically evaluate a variety of LLMs and lightweight adaptation methods, achieving the best performance by pretraining on clinical text and performing parameterefficient fine-tuning with LoRA (Hu et al., 2021). Generated impressions showcase the effectiveness of lightweight adaptation strategies for RRS.
• We investigate the impact of few-shot prompting by conducting ablation studies on the number of in-context examples provided to each model. Our findings reveal that increased context leads to improved performance across almost all cases, shedding light on the value of prompt engineering when adapting LLMs for RRS.
• We evaluate “out-of-distribution” (OOD) model performance and specifically examine the model’s ability to generalize to different imaging modalities and anatomies. Our results indicate that anatomy plays a more crucial role than modality, and best performance is achieved when training on a larger dataset which encompasses all modalities and anatomies.
449
• We conduct a reader study with radiologists who provide qualitative insights and quantitative scores on the model’s correctness, coherence, and ability to capture critical information. While our results are promising, we emphasize the need for further improvements and evaluation before clinical deployment.
Overall, our research presents a comprehensive investigation of lightweight strategies for domain adaptation in the context of RRS, offering insights into the effectiveness of different approaches and highlighting the potential of LLMs for this task. Our findings contribute toward the advancement of applied natural language processing (NLP) to radiology with implications for improving radiologists’ workflow and patient care.

heading: 2 Related Work
text: In recent years, transformer-based (Vaswani et al., 2017) language models have become ubiquitous in NLP due to their state-of-the-art performance across many tasks including language generation, question answering, and machine translation. Transformer models BERT (Devlin et al., 2018) and GPT-2 (Radford et al., 2019) established a new paradigm of first training on large amounts of general data and then fine-tuning on domain-specific data, as opposed to direct training on domainspecific data. This has led to training transformers with more parameters on increasingly more data, resulting in LLMs such as GPT-3 (Brown et al., 2020), PaLM (Chowdhery et al., 2022), and the “text-to-text transfer transformer,” or T5 (Raffel et al., 2020).
However, end-to-end fine-tuning LLMs like GPT-3 (175B parameters) requires substantial computational resources, creating a high barrier to entry. As a result, there has been a growing interest in lightweight methods for domain adaptation. One such method is prompting, in which one provides initial text input to the LLM so it has context for the given task. Performance depends heavily on the provided prompt (Brown et al., 2020), motivating principled prompting methods. Various works have pursued this in the form of natural language instructions (Liu et al., 2023; Wei et al., 2022) or supplying “in-context” examples of desired output (Lampinen et al., 2022). An alternative approach consists of parameter-efficient fine-tuning, where
one freezes existing model weights and inserts a small number of tunable parameters (Rebuffi et al., 2017; Houlsby et al., 2019; Lin et al., 2020). We focus on the two highest regarded parameter-efficient fine-tuning methods: prefix tuning (Li and Liang, 2021; Lester et al., 2021) and LoRA (Hu et al., 2021), discussed further in Section 3.
In addition to methods for prompting and finetuning, previous work has demonstrated adaptation to the medical domain via pretraining on biomedical or clinical text. Consider SciFive (Phan et al., 2021), Clinical-T5 (Lehman and Johnson, 2023), and Med-PaLM (Singhal et al., 2022), which leveraged LLMs for text generation on medical tasks. Both SciFive (biomedical) and Clinical-T5 (clinical) achieved state-of-the art results for their respective domains in tasks such as named entity recognition, natural language inference, and questionanswering. Additionally, Med-PaLM achieved success aligning text generation models to clinical tasks via methods such as chain-of-thought prompting, prompt tuning (Lester et al., 2021), and instruction tuning (Chung et al., 2022a).
Within the clinical domain, we focus specifically on the task of RRS. Previous work has approached this task with a focus on consistency and factual correctness (Zhang et al., 2020; Dai et al., 2021; Miura et al., 2021; Delbrouck et al., 2022a). To the best of our knowledge, this work is the first to leverage lightweight domain adaptation strategies on LLMs for the RRS task. Additionally, most prior work on RRS uses only chest x-rays (Dai et al., 2021; Abacha et al., 2021) from large datasets like MIMIC-CXR (Johnson et al., 2019a) and CheXpert (Irvin et al., 2019). In contrast, our work employs the MIMIC-III dataset (Johnson et al., 2016), which contains longer radiology reports from a diverse set of two imaging modalities (MR, CT) and seven anatomies (head, chest, etc.), presenting a more difficult summarization task.

heading: 3 Methods
text: As depicted in Figure 1, we investigate adapting LLMs to the task of RRS along two axes: (1) models pretrained on various combinations of natural language, biomedical, and clinical text data, as described in Section 3.1, and (2) various methods for discrete prompting and parameter-efficient finetuning, as described in Section 3.2.

heading: 3.1 Pretrained Models
text: To mitigate variance introduced by different model architectures, we focus this study on T5, i.e. “textto-text transfer transformer,” a highly regarded encoder-decoder architecture available for public use (Raffel et al., 2020). T5’s text-to-text framework enables the model to be used on any NLP task, and its pretraining over the C4 (Colossal Clean Crawled Corpus) dataset (Raffel et al., 2020) enables excellent performance in transfer learning. We include results for two architecture sizes: base (223M parameters), and large (738M parameters). From hereon we refer to T5 as the original model pretrained on C4 alone. The remaining four models are simply a version of T5 that was subsequently end-to-end fine-tuned on datasets of various relevance to the RRS task:
• FLAN-T5 (Chung et al., 2022b) tuned via instruction prompt tuning.
• SCIFIVE (Phan et al., 2021) tuned on the biomedical text dataset of PubMed, i.e. Pubmed Abstract (NCBI, 1996) and PubMed Central (NCBI, 2000).
• CLIN-T5-SCI or Clinical-T5-Sci (Lehman and Johnson, 2023) tuned on PubMed and two clinical text datasets (MIMIC-III (Johnson et al., 2016) and MIMIC-IV (Johnson et al., 2020a)).
• CLIN-T5 or Clinical-T5 (Lehman and Johnson, 2023) tuned on MIMIC-III and IV alone.
Please see the top row in Figure 1 for a visual illustration of these five models. We acknowledge the difficulty of ranking pretraining datasets’ relevance for a particular downstream task. In the case of RRS, it may seem reasonable to assert that clinical text is more relevant than biomedical text, and that biomedical text is more relevant than general natural language text. However, it becomes more difficult to compare FLAN-T5’s instruction tuning, which drastically improves performance on prompting benchmarks but has not yet been explored for medical tasks (Longpre et al., 2023). We also note the complication introduced by CLIN-T5-SCI and CLIN-T5 being trained on a subset of MIMIC-III, the same dataset used for evaluation; please see Section 5.4 for further discussion.

heading: 3.2 Lightweight task adaptation methods
text: For each pretrained model discussed in Section 3.1, we evaluate five lightweight domain adaptation methods for prompting and tuning (Fig. 1, bottom row). Prompting provides adaptation by simply supplying particular tokens to the frozen pretrained model. In contrast, parameter-efficient fine-tuning provides adaptation by adding a small number of parameters to the model and optimizing them for the task, while the original model parameters remain frozen. Compared to updating all model parameters via end-to-end fine-tuning, parameterefficient methods require much less computation and training data. We describe these five methods below in order of increasing adaptation to the downstream RRS task:
Table 1: We employ parameter-efficient fine-tuning methods for domain adaptation that modify <0.4% of model parameters while keeping other parameters frozen.
Tunable parameters Training time (hr) Model size Method # % of total per epoch total # epochs Base (223M) prefix tuning 0.37M 0.17% 0.98 9.83 10 LoRA 0.88M 0.39% 1.32 6.60 5 Large (738M) prefix tuning 0.98M 0.13% 2.93 29.3 10 LoRA 2.4M 0.32% 3.85 19.3 5
1. Null prompting (Zhao and Schütze, 2021) is a simple discrete (sequence of real naturallanguage tokens), zero-shot prompt. We supply the radiology report findings section and the basic prompt, “impression:”.
2. Prefixed prompting (Zhao and Schütze, 2021) is a discrete, zero-shot prompt with a brief instruction prepended to the original null prompt above. For our instruction we use “summarize the following radiology report:”. This provides the model some context for the RRS task. We note that a slight modification to the prepended instruction may significantly change the generated output for an individual sample. However, that same modification does not meaningfully alter quantitative metrics when applied over the entire dataset.
3. In-context learning (Lampinen et al., 2022) is a type of discrete, few-shot prompt. We begin with the null prompt and prepend one, two, or four task examples using the same template. Particular examples are chosen by computing the k-nearest neighbors (Johnson et al., 2019b) of the findings section for each training example in the embedding space of a PubMedBERT model (Deka et al., 2022). This provides the most relevant examples for the RRS task.
4. Prefix tuning (Li and Liang, 2021) is a parameter-efficient fine-tuning method which prepends and optimizes an additional taskspecific vector called the prefix as input to the model. For our base and large architectures, this requires tuning a mere 0.17% and 0.13% of total parameters, respectively (see Table 1). This approach provides the model a task-specific prompt that is very well aligned to the downstream task.
5. LoRA (Hu et al., 2021), or low-rank adapta-
tion, approximates the fine-tuning process by injecting trainable rank decomposition matrices into each architecture layer. Compared to prefix tuning, which requires some portion of the input sequence for adaptation, LoRA beneficially preserves the entire sequence length for the downstream task. For our base and large architectures, this requires tuning a mere 0.39% and 0.32% of total parameters, respectively (see Table 1). Because LoRA modifies slightly more parameters than prefix tuning, we characterize this method as having greater domain adaptation.

heading: 4 Experiments

heading: 4.1 Data
text: Our primary dataset is MIMIC-III (Johnson et al., 2016), which contains 79,790 radiology reports across two imaging modalities and seven anatomies. Table 2 contains a dataset overview. Figure 2 contains an example of a CT head report; recall the task of RRS is to predict the report impressions
section (label) given the findings section (input). We also provide some evaluations on a secondary radiology report dataset, MIMIC-CXR (Johnson et al., 2019a). This is an easier summarization task as it contains one modality and anatomy (chest xrays) with generally shorter impression sections than MIMIC-III. PhysioNet (Johnson et al., 2020b) and ViLMedic (Delbrouck et al., 2022b) provided access to pre-processed versions of these datasets, removing confidential patient information. We conduct additional processing according to domain adaptation strategies discussed in Section 3.2.

heading: 4.2 Evaluation
text: For quantitative evaluation of our generated impressions, we employ common summarization metrics such as BLEU and ROUGE-L. Between a given pair of reference and generated text, BLEU evaluates overlap using a weighted average of 1- to 4-gram precision, and ROUGE-L evaluates the longest common subsequence overlap. Beyond these token-level syntactic similarity metrics, we employ by also using metrics like BERTScore (via HuggingFace), which computes the semantic similarity between the reference and generated texts using BERT embeddings (Zhang* et al., 2020). Lastly, following previous work (Delbrouck et al., 2022a), we evaluate our model with F1-RadGraph, a F-score style metric that measures the factual correctness, consistency and completeness of generated radiology reports compared to the reference. F1-RadGraph uses RadGraph (Jain et al., 2021), a graph dataset of entities and relations present in
radiology reports. For qualitative evaluation, we include a reader study (Figure 3) with three board-certified radiologists. Each evaluated a randomly selected group of twenty generated impressions in comparison to reference impressions. They responded with either 0 (“no”), 5 (“somewhat”), or 10 (“yes”) to the following three questions:
1. Does the generated impression capture critical information? Consider if the model missed any important findings.
2. Is the generated impression factually correct? Given the text that the model did generate, consider its correctness.
3. Is the generated impression coherent, i.e. do you find the syntax comprehensible?

heading: 4.3 Experimental details
text: No further model tuning was needed for the null, prefix, and in-context discrete prompting experiments; we simply run inference with the preexisting LLM. When tuning model hyperparameters for prefix tuning and LoRA, we discovered that the same set of hyperparameters achieves best performance across all five models. This seems reasonable given that each model employs the same architecture. As such we tuned each model with the same set of hyperparameters. Please see Section A.1 in the appendix for details.

heading: 5 Results and Discussion

heading: 5.1 Domain Adaptation
text: A recurring theme throughout our results is that increased domain adaptation leads to improved performance. One axis for domain adaptation is the relevance of pretraining data to the evaluation task, in our case RRS on the MIMIC-III dataset. Figure 4 (left) demonstrates that pretraining the same architecture on increasingly relevant data (T5, SCIFIVE, CLIN-T5-SCI, CLIN-T5) typically improves performance for prefix tuning and LoRA. We do not include FLAN-T5 in this portion of the analysis, as its relative degree of domain adaptation is difficult to assess.
The second axis for domain adaptation is the amount of context provided either via longer discrete prompts (null, prefix, in-context prompting), or parameter-efficient fine-tuning (prefix tuning, LoRA). For all models, fine-tuning significantly outperforms any discrete prompting technique. Discrete prompting is still useful, however, especially if one only has black-box access to the LLM. Therefore, we compare techniques for discrete prompting in Figure 4 (right), demonstrating that prompting the model with more in-context examples improves performance for almost every model with the lone exception of T5. As T5 is the least domain-adapted model, perhaps a small number of in-context examples may actually hurt performance when examples are sufficiently out-of-domain. We leave this question to future work. Another component of Figure 4 (right) is that the remaining four models improve at different rates: SCIFIVE seems to level off near four in-context examples, while the others continue to steadily increase. This supports our hypothesis that instruction prompt tuning and maximal domain adaptation (in this case, to clinical text) improve results seen with in-context learning. This also motivates future work in instruction prompt tuning a model using domain-specific biomedical or clinical instructions.
Table 5 in the appendix includes an ablation of all configurations for domain adaptation using a base architecture (223M parameters). We subsequently take the best model (CLIN-T5) for the best methods (prefix tuning, LoRA) and scale to a large architecture (738M parameters) in Table 3. This scaling provides a significant performance boost. Similar to the base architecture, LoRA outperforms prefix tuning in all cases. Interestingly, CLIN-T5-
BASE + LoRA achieves nearly equivalent performance to CLIN-T5-LARGE + prefix tuning, exemplifying the benefits of LoRA over prefix tuning. Finally, after performing all prior analysis using the MIMIC-III dataset, we apply the best combination (CLIN-T5-LARGE + LoRA) on MIMIC-CXR and include results in Table 3.

heading: 5.2 Out-of-distribution performance
text: Table 4 demonstrates out-of-distribution performance using a CLIN-T5-BASE model prefix tuned on CT head data. When evaluating on a different test set, the model better summarizes reports on a different modality (MR head) than a different anatomy (CT other). This suggests that report finding tokens are more anatomy- than modalityspecific, which seems reasonable. Counterintuitively, the model performs worse when shifting just the anatomy (CT other) compared to shifting the modality and anatomy (MR other). This could be due to a myriad of reasons, such as MR findings being 15% longer than CT findings and MR other containing fewer anatomies (four) than CT other (five). Lastly, we find that training CLIN-T5BASE on all data leads to higher performance than training on CT head alone.

heading: 5.3 Qualitative Evaluation

heading: 5.3.1 Error Analysis
text: We perform a qualitative error analysis of 20 randomly selected reference findings, reference impressions, and generated impressions via CLINT5-LARGE on the CT head dataset. Please see Figure 2 for an example. We describe four types of deviations from the reference impressions and their corresponding colors used in Figure 2:
1. Factually correct text that is novel compared to the reference impression (green). This text is present in the reference findings but not in the reference impression.
2. Incoherent but potentially relevant text (orange). This contains medically relevant information that is also included in the reference, but is presented with incoherent grammar.
3. Hallucinated filler text (pink). This includes extra punctuation or common words such as “findings.” These are filler text because they are undesirable but do not detract from the correctness of the generated impression.
4. Hallucinated medical text (red). This includes text that is either (1) not explicitly included
in the findings or reference impression, or (2) relevant to the findings but communicated in a factually incorrect manner. This is the worst of the four deviations, as we want the model to avoid inferring information which didn’t originate directly from the radiologist.

heading: 5.3.2 Reader study
text: Three radiologists evaluated impressions generated via CLIN-T5-LARGE + LoRA on the CT head dataset according to the procedure described in Section 4.2. Results in Figure 3 are encouraging, as our generated impressions scored well for all three questions. Additionally, the radiologists shared the following observations:
• Occasionally there is extra information included in the “reference” impression that is not available in the findings, i.e. which the model has no chance of summarizing.
• The model may include duplicate or resummarized when referring to prior studies. For example, the reference will state “area of subarachnoid hemorrhage ... which is unchanged since the patient’s prior scan,” while the generated impression merely says “no significant change since the prior study.” This difference is typically an institutional or personal preference.
• The model made an incorrect reference to the patient’s prior medical history. The reference was “subtle hypodensity in the left frontal lobe. given lack of prior studies available for comparison,” but the generated impression was “subtle hypodensity ... in the left frontal lobe, which is consistent with prior studies.”
This study provided valuable insights which could not be obtained via quantitative metrics. Fundamentally we advocate for the use of reader studies when evaluating report summarization to facilitate more clinically relevant research.

heading: 5.4 Pitfalls
text: We now discuss weaknesses in our analysis which motivate future study. One potential concern is that CLIN-T5 and CLIN-T5-SCI were pretrained over the MIMIC-III dataset. This serves our purpose for evaluating models with high levels of domain adaptation, but it could result in data leakage if Lehman and Johnson (2023) used the test set for pretraining.
We also evaluate each model on a second clinical dataset, MIMIC-CXR, and CLIN-T5 similarly performs the best (Table 6, appendix).
Ultimately though, while we chose CLIN-T5 in Table 3 because it achieves the highest scores, we note the comparable performance of models which did not pretrain with MIMIC-III, such as T5, FLAN-T5, and SCIFIVE (see Appendix Table 5).
Another weakness is that “domain” and “distribution” are not rigorously defined. Our intuitive characterizations could be improved by quantifying or visualizing the distance between different distributions using various embedding- or graph-based methods (Johnson et al., 2019b; Jain et al., 2021).
Lastly, we made assumptions determining the best configuration of model and prompting method. We first performed a comprehensive evaluation of all models and methods using the base architecture (223M parameters) on MIMIC-III (Table 5, appendix) and all models with LoRA on MIMICCXR (Table 6, appendix). From these results we chose only the best model (CLIN-T5) for scaling to the large architecture (738M parameters) due to compute constraints. Future work should include a comprehensive evaluation of all configurations on both architecture sizes and datasets.

heading: 6 Conclusion
text: Our research employs innovative lightweight strategies to adapt LLMs for the task of RRS. We investigate how domain adaptation—both via model pretraining on relevant data and via methods for discrete prompting and parameter-efficient fine-tuning—affects downstream RRS task performance. We achieve best performance using a model pretrained on clinical text (CLIN-T5) and subsequently fine-tuned with RRS samples using LoRA. These compelling results require tuning a mere 0.32% of model parameters. While further validation is required before clinical deployment, we believe our findings contribute to the literature and advance the potential for improved radiologist workflows and patient care.

heading: 7 Acknowledgements
text: Our research is a direct continuation of the radiology report summarization track conducted at ACL BioNLP 2023 (Delbrouck et al., 2023).

heading: A Appendix
text: A.1 Hyperparameters for parameter-efficient fine-tuning
For the prefix tuning experiments, we tune each LLM with the following hyperparameters:
• Initial learning rate of 1e−2 that linearly decays to 1e−3 after a 100-step warm-up.
• Ten epochs maximum with an early stopping criterion if validation loss has not decayed for five consecutive epochs.
• Batch size of eight (large architecture) or 16 (base architecture) with four gradient accumulation steps, rendering an effective batch size of 32 or 64, respectively.
For the LoRA experiments, we tune each LLM with the following hyperparameters:
• Initial learning rate of 1e−3 that decays linearly to 1e−4 after a 100-step warm-up.
• Five epochs with no early stopping criterion.
• Batch size of six with four gradient accumulation steps, rendering an effective batch size of 24.
LoRA requires slightly more memory than prefix tuning, hence we adjusted the effective batch size to comfortably fit on our NVIDIA Quadro RTX 8000 GPU. Despite the larger memory footprint and greater training time per epoch (Table 1), LoRA requires fewer epochs to reach convergence than prefix tuning, resulting in 30% less tuning time overall. We note the importance of a learning rate warm-up over the first 100 gradient steps, which has been shown beneficial for low-data settings (Li and Liang, 2021). We experimented with various learning rate schedulers (step, exponential decay) but found linear decay to give slightly better performance in terms of validation loss. As discussed in Section 4.3, the best set of hyperparameters is constant across each of the five models.

heading: Future Work
text: We now discuss weaknesses in our analysis which motivate future study. One potential concern is that CLIN-T5 and CLIN-T5-SCI were pretrained over the MIMIC-III dataset. This serves our purpose for evaluating models with high levels of domain adaptation, but it could result in data leakage if Lehman and Johnson (2023) used the test set for pretraining. We also evaluate each model on a second clinical dataset, MIMIC-CXR, and CLIN-T5 similarly performs the best (Table 6, appendix). Ultimately though, while we chose CLIN-T5 in Table 3 because it achieves the highest scores, we note the comparable performance of models which did not pretrain with MIMIC-III, such as T5, FLAN-T5, and SCIFIVE (see Appendix Table 5). Another weakness is that “domain” and “distribution” are not rigorously defined. Our intuitive characterizations could be improved by quantifying or visualizing the distance between different distributions using various embedding- or graph-based methods (Johnson et al. , 2019b; Jain et al. , 2021). Lastly, we made assumptions determining the best configuration of model and prompting method. We first performed a comprehensive evaluation of all models and methods using the base architecture (223M parameters) on MIMIC-III (Table 5, appendix) and all models with LoRA on MIMICCXR (Table 6, appendix). From these results we chose only the best model (CLIN-T5) for scaling to the large architecture (738M parameters) due to compute constraints. Future work should include a comprehensive evaluation of all configurations on both architecture sizes and datasets