WisPerMed at BioLaySumm: Adapting Autoregressive Large Language Models for Lay Summarization of Scientific Articles (2024)

Abstract: This paper details the efforts of the WisPerMed team in the BioLaySumm2024 Shared Task on automatic lay summarization in the biomedical domain, aimed at making scientific publications accessible to non-specialists. Large language models (LLMs), specifically the BioMistral and Llama3 models, were fine-tuned and employed to create lay summaries from complex scientific texts. The summarization performance was enhanced through various approaches, including instruction tuning, fewshot learning, and prompt variations tailored to incorporate specific context information. The experiments demonstrated that fine-tuning generally led to the best performance across most evaluated metrics. Few-shot learning notably improved the models’ ability to generate relevant and factually accurate texts, particularly when using a well-crafted prompt. Additionally, a Dynamic Expert Selection (DES) mechanism to optimize the selection of text outputs based on readability and factuality metrics was developed. Out of 54 participants, the WisPerMed team reached the 4th place, measured by readability, factuality, and relevance. Determined by the overall score, our approach improved upon the baseline by ≈ 5.5 percentage points and was only ≈ 1.5 percentage points behind the first place.

text: Proceedings of the 23rd Workshop on Biomedical Language Processing, pages 769–779 August 16, 2024. ©2024 Association for Computational Linguistics
769

heading: 1 Introduction
text: In the biomedical domain, scientific publications and research play a central role in communicating research findings and results. However, these documents are usually written in complex language and use terminology and technical jargon that can be challenging for lay readers or researchers from different fields to understand (Goldsack et al., 2022). In this context, lay summarization can be utilized to extract the most relevant information from the original article or publication while also providing
supplementary explanations. This often entails incorporating background information that may not be contained within the article itself.
In this context, this paper presents the participation of the team WisPerMed in the BioLaySumm2024 Shared Task (Goldsack et al., 2024) on automatic lay summarization and describes the employed approaches to tackle this challenge.
Summaries generated by LLMs, as demonstrated by Zhang et al. (2024), can be of equivalent or superior quality to original references. Additionally, instruction tuning is an effective approach for enhancing performance. However, LLMs face limitations when applied to domain-specific abstractive summarization. Key challenges include the quadratic complexity of transformer-based models (Vaswani et al., 2017) concerning input text length, model hallucination, where factually incorrect text is generated, and domain shift from training to test data (Afzal et al., 2023). Similarly, studies on text simplification (Amin et al., 2023) indicate that although general-purpose LLMs are capable of effectively simplifying clinical reports, they sometimes generate factual inaccuracies and omit crucial information.
To adapt LLMs to a specific domain or task (Ling et al., 2024), it is possible to fine-tune the models, leverage few-shot learning or further pre-train the models on domain data. Examples of domainadapted LLMs for the biomedical domain include the BioMistral (Labrak et al., 2024) and OpenBioLLM (Pal and Sankarasubbu, 2024) model series. The BioMistral models are based on the Mistral 7B Instruct v0.1 (Jiang et al., 2023) model. They are further pre-trained on the PMC Open Access Subset1. OpenBioLLM models are based on the Llama3 (AI@Meta, 2024) models and were
1https://www.ncbi.nlm.nih.gov/pmc/tools/ openftlist/ Accessed: 2024-05-17
adapted to the biomedical domain through finetuning.

heading: 2 Dataset
text: The dataset (Goldsack et al., 2022) of the Shared Task (Goldsack et al., 2024) contains two collections of scientific journal articles and the corresponding lay summaries, namely PLOS and eLife. PLOS and eLife also include the section headings and keywords of the article. The PLOS dataset has 24,773 examples in the training split and 1,376 examples in the validation split, whereas the eLife dataset is smaller with 4,346 examples in the training split and 241 examples in the validation split. The test split consists of 142 examples for both datasets. Lay summaries of the PLOS dataset were written by the authors of the articles and are approximately 150-200 words long, while eLife lay summaries were written by expert editors in correspondence with the authors and are about twice as long.
For the remainder of this paper, any reference to the validation or test set will include eLife and PLOS unless otherwise specified.

heading: 3 Evaluation Metrics
text: The generated summaries were evaluated across ten metrics that fall into the following categories: relevance, readability, and factuality. Relevance was assessed through Recall-Oriented Understudy for Gisting Evaluation (Lin, 2004) (ROUGE-1, ROUGE-2, ROUGE-L) and BERTScore (Zhang et al., 2020). ROUGE counts the overlapping ngrams in the generated texts and target lay summaries, whereas BERTScore uses contextual word embeddings to compare the semantic similarity of the two texts. Readability was evaluated using the Flesch-Kincaid Grade Level (FKGL) (Kincaid, 1975), Dale-Chall Readability Score (DCRS) (Chall and Dale, 1995), Coleman-Liau Index (CLI) (Coleman and Liau, 1975), and Learnable Evaluation Metric for Text Simplification (LENS) (Maddela et al., 2023). The FKGL uses sentence lengths and syllable count per word to estimate readability. The DCRS uses a word list to compute the occurrences of words unknown to most 4-th graders and the CLI estimates the grade level necessary to comprehend the text. The LENS metric is a learnable evaluation metric trained on datasets containing human ratings of simplifications. In this setting, LENS measures the simplification of the
abstract by the generated text using the target lay summary as a reference. Factuality was assessed with AlignScore (Zha et al., 2023) and Summary Consistency (SummaC) (Laban et al., 2022). The AlignScore quantifies the degree of alignment between the facts in the summary and the scientific article, while SummaC also includes consistency.

heading: 4 Methods and Experiments
text: This section outlines the methodology employed in the experiments conducted on the specified dataset.

heading: 4.1 Fine-tuned Models
text: In this study, instruction tuning (Wei et al., 2022) was utilized to fine-tune various models. Instruction tuning refers to the process of fine-tuning language models on a collection of datasets described via instructions. BioMistral-7B-DARE (BioM) and Llama3-70B-Instruct (Llama3) were fine-tuned for one epoch utilizing Quantized Low-Rank Adaptation (QLoRA) (Dettmers et al., 2023) on the eLife and PLOS dataset individually. BioM was trained on the abstracts + lay summaries, whereas Llama3 was trained on the entire articles + lay summaries. The texts were structured using the Mistral and Llama3 instruction templates prior to the finetuning process. Please refer to the Appendix A, B, and C for details on prompts, parameters and licenses, respectively.
After evaluating the checkpoints of BioM on the validation set, the checkpoints with the best scores were selected for inference. For Llama3, the final checkpoints were selected. The models were given the same prompt as during fine-tuning but without the target.

heading: 4.2 Prompt Variations
text: Prompts can guide the LLM’s content generation process without the need for fine-tuning. In the zero- and few-shot settings, different prompt variations and their effect on the evaluation metrics were examined. In the few-shot setting, example lay summaries from the training and validation set were included in the prompt when performing inference on the validation and test set, respectively. The format of these few-shot prompts is designed to emulate a preceding conversation with the model, with the included examples serving as the model’s previous responses.
To choose the best few-shot examples, all examples were ranked based on their average nor-
malized readability and factuality. The two and three highest-ranked examples were selected for the eLife and PLOS datasets, respectively.
An initial prompt was created by replicating the prompt used for inference with the fine-tuned BioM model (see Appendix A). This prompt was then tested with BioM and OpenBioLLM-70B (OpenBio) on the validation set.
Additionally, three prompt variations were created, which provide the model with different kinds of context information. It was decided that BioM would be utilized for all experiments involving these variations due to its superior performance on the validation set in the few-shot setting (see Table 3 in Appendix D). LLMs can assume different roles and adapt their vocabulary accordingly (Salewski et al., 2023), resulting in enhanced performance in tasks related to the specified role. Accordingly, the first prompt variation comprises a persona description of a science communicator (BioMpers), instructing the model to utilize the expertise of this persona to create the lay summary based on the abstract. The model is then instructed to channel the expertise of the described persona to craft the lay summary based on the abstract. The second prompt variation is a modification of the initial prompt, incorporating the introduction to provide additional background information because associated context can improve LLM performance (Karmaker and Feng, 2023). The second prompt variation is a modification of the initial prompt, but it includes the
introduction as further context for background information (BioMintro). The third prompt variation includes the abstract and a guide on how to write a lay summary (BioMguide), accompanied by instructions concerning the content and style of the requested summary. This method leverages the importance of clear and detailed task directives. The selection of these prompts was based on a few preliminary experiments with the model and an initial assessment of the responses. However, no comprehensive optimization was performed. The wording of all prompts can be found in Appendix A.
Due to the efficacy of few-shot learning with the initial prompt, the prompt variations were implemented in a few-shot setting on the test set.

heading: 4.3 Dynamic Expert Selection (DES)
text: The success of an LLM depends on factors such as the properties of the dataset, the complexity of the domain, and the design of the prompt (Ling et al., 2024). Consequently, a model may yield a more suitable lay summary when prompted in a different manner. In addition, the output quality depends upon the selection of the inference parameters (Minaee et al., 2024). In consideration of this assumption, a Dynamic Expert Selection (DES) was developed. It selects the most appropriate text from a set of candidate texts based on metrics that do not require a reference lay summary.
The mechanism uses the readability metrics FKGL, DCRS, and CLI, as well as the factuality
metrics AlignScore and SummaC. These metrics are computed for each candidate text. Readability scores are multiplied by -1 so that higher scores indicate better readability. All scores are normalized using min-max normalization to range between 0 and 1, where 1 is the best and 0 is the worst. For each candidate text, an overall score is calculated by multiplying the means with different weights. Given that the target lay summaries in eLife have a higher readability than those in PLOS (Goldsack et al., 2022), the overall scores are computed with different weights for the two aspects. For eLife summaries: Readability is weighted at 0.675 and factuality at 0.325. For PLOS summaries: Readability is weighted at 0.25 and factuality at 0.75. The candidate text with the highest overall score is selected as the most suitable lay summary. The selection of the weights is based on the assumptions about the target texts and comparisons of the overall scores on the validation dataset.
This approach was applied to BioM in the fewshot setting using all prompt variants (see Figure 1) and to the fine-tuned BioM using two distinct inference parameter settings (see Appendix B).

heading: 5 Results
text: The results of the experiments using BioM, Llama3, and OpenBio are presented in table 1. The experiments are categorized into zero-shot learning, fewshot learning, and fine-tuning.
BioM exhibits the highest LENS score in the zero-shot setting. However, its relevance and factuality performance are the lowest. Few-shot learning resulted in enhanced performance across all metrics except for LENS. The persona prompt (BioMpers) led to an improvement in relevance. Including the introduction in the prompt (BioMintro) resulted in a reduction in all aspects despite the fact that the model had access to more information from the article itself. In comparison, the prompt with the guide (BioMguide) exhibits minimal enhancements. The optimal few-shot learning for BioM occurred with the initial prompt, which achieved the highest readability and factuality in the few-shot setting, excluding the DES approach. However, OpenBio slightly underperformed with this prompt in the few-shot setting, except for the LENS score, where it performed best in this setting.
The DES used all four prompts and outperformed the baseline with improvements in factuality and readability, achieving the best results in the
few-shot setting. Fine-tuning BioM improved relevance and factuality scores, though the LENS score decreased slightly, with other readability metrics similar to the few-shot setting. The fine-tuned BioM outperformed the baseline in terms of relevance and overall quality. The DES approach improved all metrics except for a slight drop in the LENS score. In contrast, Llama3 underperformed despite being larger. It was less effective at extracting relevant information from full articles and produced lowerquality text in terms of readability, even though its LENS score was higher than BioM’s. Additionally, Llama3’s factuality scores decreased, leading to an overall performance drop compared to the baseline.

heading: 6 Conclusion
text: This paper presents the WisPerMed team’s approaches to automatic lay summarization within the biomedical domain, utilizing a combination of fine-tuning, prompt variations, and Dynamic Expert Selection.
Among these approaches, fine-tuning emerged as an effective method, leading to the best performance across most metrics. This underscores the importance of task-specific training in optimizing model output for complex summarization tasks. Additionally, BioM showed strong few-shot learning capabilities, illustrating its robustness and versatility in generating accurate and relevant summaries even without extensive training. As the model adjusts to the factuality and readability of given examples, providing better examples could lead to further enhancements in these aspects.
BioM reached high factuality, even when provided solely with abstracts as input, suggesting that BioM leveraged domain-specific knowledge acquired during pre-training. This indicates that domain adaptation remains an important factor when using LLMs for lay summarization of scientific articles, as BioM outperformed the larger general model Llama3.
The four prompt variations exhibited differing effects on the evaluation metrics. BioM is adept in fulfilling the role of a science communicator (BioMpers), as evidenced by the enhanced relevance. BioMintro and BioMguide did not significantly enhance the metrics, indicating that the increase in context was not beneficial for all texts. Without DES, a shorter prompt (BioMinitial) yielded the optimal results, suggesting that the model effectively
comprehends the task from the provided examples. The DES mechanism further refined readability and, in particular, factuality by retrospectively selecting the best text outputs based on evaluation metrics. This highlights the potential of metricdriven selection to improve the quality of lay summaries further.
In conclusion, our study demonstrates that finetuning, the use of informed prompt variations, and selection mechanisms can enhance the capability of autoregressive LLMs to produce lay summaries that are factually accurate, relevant, and readily accessible to non-specialist audiences. This approach fosters broader public engagement with scientific findings, advancing the goal of making biomedical research comprehensible and accessible.
Limitations
Only four discrete prompts in combination were tested with DES, and only two sets of inference parameters were explored. This limited scope means that the findings may not fully capture the potential variability and performance of the the various models under different conditions. The weights for the Dynamic Expert Selection method were chosen based on heuristics without any formal op-
timization, which could impact the robustness and generalizability of the results. Another limitation is the possibility that BioM may have been previously exposed to the gold standard summaries. If this is the case, it could skew the results by artificially inflating the model’s performance. These limitations indicate potential avenues for future research, including the necessity for more comprehensive prompt engineering, optimization of DES weights, and a wider range of tasks to ensure the robustness of the approach. Another potential future direction is adapting these methods for other complex domains or languages and exploring additional metrics.

heading: Acknowledgement
text: The work of Tabea M. G. Pakull, Hendrik Damm, Ahmad Idrissi-Yaghir and Henning Schäfer was funded by a PhD grant from the DFG Research Training Group 2535 Knowledge-and data-based personalisation of medicine at the point of care (WisPerMed).

heading: A Prompts
text: The prompts used in the experiments are shown in Figures 2, 3, 4, 5, and 6.

heading: B Setup and Hyperparameter
text: Training All trainings were executed on a single Nvidia H100 80GB using the unsloth2 framework and QLoRA (Dettmers et al., 2023). The following modules were targeted with QLoRA: “q_proj”, “k_proj”, “v_proj”, “o_proj”, “gate_proj”, “up_proj”, and “down_proj”. The QLoRA rank and alpha were both set to 16. The QLoRA dropout was set to 0. The optimization of the models was conducted using the 8-bit Adam optimizer (Loshchilov and Hutter, 2019), which was configured with a maximum learning rate of 2× 10−4 and a weight decay factor of 0.01. The learning rate schedule included a linear decay following an initial phase consisting of five warm-up steps. Maximum sequence length was set to 4,096.
Inference For the inference process, a greedy search algorithm was employed as the decoding strategy (Minaee et al., 2024), with a configuration that allowed for the generation of up to 1024 new tokens per inference iteration.
DES The DES with the fine-tuned model used the inference parameter as described above for one candidate, and a repetition penalty of 1.1 was chosen to generate another candidate.

heading: C Licenses
text: In Table 2 the Licenses as given by the owners of the Framework/Model are displayed.
2https://github.com/unslothai/unsloth Accessed: 2024-05-17 3https://github.com/unslothai/unsloth Accessed: 2024-05-17 4https://huggingface.co/BioMistral/BioMistral-7B-DARE Accessed: 2024-05-17 5https://huggingface.co/meta-llama/Meta-Llama-3-70B-Instruct Accessed: 2024-05-17 6https://huggingface.co/aaditya/Llama3-OpenBioLLM-70B Accessed: 2024-05-17

heading: D Results on the Validation Set
text: The results of experiments on the validation set and the reference scores of the target lay summaries and input abstracts are presented in Table 3.

heading: Future Work
text: These limitations indicate potential avenues for future research, including the necessity for more comprehensive prompt engineering, optimization of DES weights, and a wider range of tasks to ensure the robustness of the approach. Another potential future direction is adapting these methods for other complex domains or languages and exploring additional metrics