HGP-NLP at BioLaySumm: Leveraging LoRA for Lay Summarization of Biomedical Research Articles using Seq2Seq Transformers (2024)

Abstract: Lay summarization aims to generate summaries of technical articles for non-experts, enabling easy comprehension for a general audience. The technical language used in research often hinders effective communication of scientific knowledge, making it difficult for nonexperts to understand. Automatic lay summarization can enhance access to scientific literature, promoting interdisciplinary knowledge sharing and public understanding. This has become especially important for biomedical articles, given the current global need for clear medical information. Large Language Models (LLMs) (Yao et al., 2024) , with their remarkable language understanding capabilities, are ideal for abstractive summarization, helping to make complex information accessible to the public. This paper details our submissions to the BioLaySumm 2024 Shared Task: Lay Summarization of Biomedical Research Articles (Goldsack et al., 2024). We fine-tune and evaluate sequence-to-sequence models like T5 across various training dataset settings and optimization methods such as LoRA (Hu et al., 2021b) for lay summarization. Our submission achieved the 53rd position overall.

text: Proceedings of the 23rd Workshop on Biomedical Language Processing, pages 831–836 August 16, 2024. ©2024 Association for Computational Linguistics
831

heading: 1 Introduction
text: Scientific research aims to advance knowledge, but a major challenge is the lack of domain-specific knowledge among the public. Technical reports and research articles are often incomprehensible to non-experts, hindering knowledge dissemination. Lay summarization addresses this by generating factual, readable summaries of technical texts for non-experts (Chaturvedi et al., 2020). In the biomedical field, lay summarization is crucial due to its highly technical content involving complex medical terms and detailed research findings.
* Dept. of Data Science & Computer Applications † Equal Contribution ‡ Dept. of Computer Science & Engineering
Access to clear medical information is essential for informed health decisions. Misunderstandings or a lack of access can lead to poor decisions, increased anxiety, and a general distrust of medical advice (Guo et al., 2021). Recent advancements in large language models (LLMs) (Yao et al., 2024) and autoregressive LLMs (Chen et al., 2023) like ChatGPT, Gemini, Mistral, and Llama have shown significant potential. With their large-scale pretraining, these models can generate high-quality, contextually relevant summaries that are informative and accessible. However, they are computationally expensive and often require fine-tuning for specific tasks, needing substantial computational resources and energy. Sequence-to-sequence models (Chiu et al., 2018) provide a promising alternative, addressing these computational challenges. With an encoder-decoder structure, these models efficiently handle input and output sequences, making them well-suited for summarization (Kouris et al., 2021). They are more computationally efficient than LLMs, requiring fewer resources while delivering high performance. This paper describes our approach to the BioLaySumm 2024 Shared Task: Lay Summarization of Biomedical Research Articles (Goldsack et al., 2024). We fine-tune and evaluate sequence-to-sequence models like T5 for the task of lay summarization (Challagundla and Peddavenkatagari, 2024) across various training dataset settings and optimization methods such as LoRA (Hu et al., 2021b). Our final submission to this task is a LoRA-based Flan-T5-Base model (Rusnachenko and Liang, 2024), which secured us the 53rd position on the leaderboard.

heading: 2 Background and Previous Works

heading: 2.1 Problem and Data Description
text: The BioLaySumm shared task at the BioNLP Workshop of ACL 2024(Goldsack et al., 2024) focuses on the abstractive summarization of biomedical ar-
ticles. The goal is to generate summaries that are accessible to non-expert audiences by including more background information and using less technical terminology, effectively creating a “lay summary.” Task Definition: Given an article’s abstract and main text, participants are required to train a model (or models) to generate a lay summary. Two separate datasets, derived from the biomedical journals PLOS and eLife, are provided for model training and will be used for evaluation. For the final evaluation, submissions will be ranked based on their average performance across both datasets.
BioLaySumm offers two datasets from wellrespected scientific journals: the Public Library of Science (PLOS) and eLife. Each dataset provides pairs of original research articles along with their corresponding summaries, written by scientists specifically for a public audience, as indicated in Table 1. The PLOS dataset is the larger of the two, containing 24,773 instances for training, 142 for testing, and 1,376 for validation. The eLife dataset contains 4,346 instances for training, 142 for testing, and 241 for validation, as shown in the Table 2. Each dataset includes train, validation, and test sets in the form of JSONL files. The eLife summaries have an average length of around 300-350 words, whereas the PLOS summaries are shorter, averaging around 160-200 words (Goldsack et al., 2022).

heading: 2.2 Automatic Text Summarization
text: Summarization involves condensing large amounts of text into key points. Extractive summarization, like underlining key passages in a textbook, extracts important sentences directly from the original text. This method is simpler and ensures factual accuracy, but the summaries may lack coherence (Neto et al., 2002). On the other hand, Abstractive summarization aims to understand and rephrase the main ideas in a new, concise way, offering more readable summaries but potentially introducing errors if the model misinterprets the original text (Gupta and Gupta, 2019). It goes beyond copy-
ing sentences, striving to grasp the core message of a text. Traditional summarization approaches typically fall into two categories: extractive methods that select and combine important sentences from the source text, and abstractive methods that generate entirely new text that captures the main ideas. This field has seen significant progress with transformer-based methods excelling at understanding complex relationships in text. Recently, large language models (LLMs) (Alberts et al., 2023) utilizing autoregressive techniques have emerged as a powerful approach, offering the ability to not only summarize but also generate creative text formats (Poornash et al., 2023).

heading: 2.3 Biomedical Lay Summarization

heading: 2.3.1 Biomedical Summary Corpora
text: In the field of deep learning, large collections of text, known as corpora (Stubbs, 2004), are crucial for training language models. These datasets help machines learn the patterns, meaning, and structure of language, thereby improving their ability to understand and generate text for tasks such as sentiment analysis or machine translation (Gilquin and Gries, 2009). Instead of creating a new corpus from scratch, focus on utilizing existing collections of scientific articles alongside their corresponding, simplified summaries written for the general public. The BioLaySumm datasets (PLOS and eLife) are built by extracting these paired texts from the corpora. These datasets then become the foundation for training and evaluating models that can automatically generate clear summaries of scientific research articles for a non-scientific audience (Goldsack et al., 2022).

heading: 2.3.2 Sequence-to-sequence models
text: Significant research has utilized sequence-tosequence (seq2seq) models (Sriram et al., 2017) for generating lay summaries of biomedical research articles. Leveraging pre-trained models like T5 and BART (Colak and Karadeniz, 2023), researchers have explored methods to effectively capture the complexities of biomedical language and translate it into clear summaries for the public. Another area
of focus involves adapting Transformers (Dandan et al., 2023) by incorporating biomedical domain knowledge, such as using pre-trained models on scientific text or knowledge graphs, to enhance the model’s understanding of scientific concepts. Additionally, research (Fan et al., 2018) investigates controllable summarization with seq2seq models, allowing researchers to tailor summaries to specific needs like factual accuracy or readability (Sriram et al., 2017).

heading: 2.3.3 Generative Pre-trained Transformers
text: Generative Pre-trained Transformers (GPT) (Zhu and Luo, 2022) are advanced auto-regressive transformer models trained on vast amounts of data, enabling them to generate human-like text and perform various natural language understanding tasks with high accuracy and efficiency (Luo et al., 2022). Their strong performance is attributed to the quality of the training dataset and their autoregressive nature, where the input serves as the "prefix" of the output. However, to maximize their performance, fine-tuning on specific downstream tasks is necessary, which can be computationally expensive.

heading: 2.3.4 Infusing External Knowledge
text: Integrating external knowledge into models (Paulheim, 2017) involves incorporating domainspecific data or structured information, which boosts performance and contextual comprehension for more accurate predictions and insights. This approach enriches deep learning models (Menghani, 2023) with additional context and expertise, enhancing their robustness and interpretability (Koncel-Kedziorski et al., 2022). One widely used technique involves integrating external knowledge graphs (Paulheim, 2017) for downstream tasks like summarization. This leverages semantic relationships to better understand and contextualize concepts within summaries (Goldsack et al., 2023).

heading: 2.4 Current State of NLP
text: State-of-the-art foundation large language models (LLMs) like ChatGPT (Firat, 2023), Gemini (Khurdula et al., 2024), and LLaMA (Touvron et al., 2023) are revolutionizing how we interact with text. These LLMs are auto-regressive in nature and can generate various creative text formats and translate languages with impressive fluency. However, they still require fine-tuning to achieve state-of-theart performance in downstream tasks. This can be done by optimizing on a downstream dataset
and task or by using methods involving prompting. Fine-tuning these models is computationally expensive due to their large number of parameters. Techniques such as Low-Rank Adaptation of Large Language Models (LoRA) (Hu et al., 2021b) and Parameter-Efficient Fine-Tuning (PEFT) (Sabry and Belz, 2023) address this challenge. These methods enable fine-tuning by freezing the pre-trained model weights and injecting trainable rank decomposition matrices into each layer of the Transformer architecture, greatly reducing the number of trainable parameters for downstream tasks.

heading: 3 System Overview
text: Our approach to the task involved fine-tuning sequence-to-sequence transformer-based models (Ramachandran et al., 2016), such as T5, across various training dataset settings and optimization methods like LoRA on the PLoS and eLife datasets. The models were evaluated based on the readability (Paasche-Orlow et al., 2003), relevance (Montague and Aslam, 2001), and factuality (Pagnoni et al., 2021) of the summaries generated. The model demonstrating the best performance across these three evaluation criteria was chosen as our final submission for the task.

heading: 3.1 Data Pre-Processing
text: Each sample in the dataset has attributes as illustrated in Table 1. These were utilized to design a prompt structured as follows:
Provide a lay summary of the following article, which includes keywords <keywords>: <article>.
For processing text, the T5 tokenizer was used with a maximum input of 1024 tokens. LoRA-based models, on the other hand, could handle up to 2048 tokens as input (Seye et al., 2018). In all cases, the output was capped at 512 token.

heading: 3.2 Fine-Tuning Transformers
text: T5 is a text-to-text Transformer architecture that treats all tasks, like translation and classification, as generating target text from input text. It differs from BERT by adding a causal decoder and using various pre-training tasks instead of the cloze task. We trained the sequence-to-sequence transformer T5 on the given datasets using different approaches. This included training solely on either the eLife or PLoS dataset, followed by inference on the evaluation and test sets. Alternatively, we also trained
the model on a combined dataset consisting of both train sets before performing inference. Flan-T5 is an extension of T5 (Rusnachenko and Liang, 2024) that has been further fine-tuned on a diverse set of instructions, enhancing its performance on various downstream tasks, including summarization. To fine-tune this model, we employed the Parameter Efficient Fine Tuning Method, specifically the Low-Rank Adaptation (LoRA) technique (Hu et al., 2021a), which efficiently fine-tunes pretrained models by adapting a small number of model parameters. This method significantly reduces computational requirements while improving performance.

heading: 3.3 Implementation Details
text: The experiments were conducted with a learning rate of 1e-3. For fine-tuning the T5-base (Guan et al., 2024) using LORA (Hu et al., 2021a), we selected a rank of 32, a LoRA alpha value of 32, and a LoRA dropout of 0.05. The T-5 small model was trained for 10 epochs while the Flan-T5 model was trained for 2 epochs. All experiments were carried out on a P100 GPU via Kaggle.

heading: 4 Analysis & Results
text: The challenge provides two individual datasets of different distributions for training, evaluation, and testing. Details about the distribution of the datasets are illustrated in Table 2. To evaluate the lay summary, we have taken into account three major factors: readability, relevance, and factuality. Readability scores, i.e., FleschKincaid Grade Level (FKGL), Dale-Chall Readability Score (DCRS), Coleman-Liau Index (CLI), and LENS assess how easy summaries are to understand for a lay audience. Relevance scores i.e., ROUGE (1, 2, and L) and BERTScore measure how well summaries capture key points from the original research. Factuality scores i.e., AlignScore, SummaC evaluate how accurately summaries reflect factual information.
The detailed results of our experiments are presented in Table 3. Although none of the models achieved high scores, this can be attributed to the lack of constraints and the limited training and context length of the models. This leads to all models converging to similar scores despite differences in model sizes and training methodologies. However, it is evident that training on both datasets together improves the readability and factuality of the models. This improvement is also observed when fine-tuning a large model using parameterefficient fine-tuning techniques like LORA (Hu et al., 2021b) , which is originally pre-trained on a large text corpus. This might be due to the fact that PEFT (Pu et al., 2023) techniques enable large language models to learn more effectively while retaining previously acquired knowledge. However, this claim for lay summarization would require further experiments to validate.

heading: 5 Conclusion
text: We fine-tune and evaluate sequence-to-sequence models like T5 for the task of lay summarization (Challagundla and Peddavenkatagari, 2024) across various training dataset settings and optimization methods such as LoRA. We extended our evaluation beyond traditional accuracy metrics to encompass real-world application considerations like relevance (Montague and Aslam, 2001) , readability (Paasche-Orlow et al., 2003) , and factuality (Pagnoni et al., 2021) . Our analysis revealed interesting trade-offs of fine-tuning pre-trained models using traditional and parameter-efficient methods like LoRA (Hu et al., 2021b). Future work should explore the impact of PEFT-similar models for lay and abstractive summarization.
Limitations
Due to computational resource limitations, we were able to conduct only a limited number of experiments and were constrained by input token limits. Our access was limited to the Kaggle P100 GPU,
for which we are grateful. This restriction led us to primarily experiment with smaller model sizes, potentially missing the benefits of larger architectures. Our hyperparameter tuning was not extensive, and as a result, our models’ performance fell short of high scores on evaluation metrics. This indicates substantial room for improvement. We focused narrowly on sequence-to-sequence models, and these limitations present clear opportunities for future research to build upon and investigate.

heading: Acknowledgments
text: We would like to thank the Research Society MIT, an undergraduate interdisciplinary technical society of Manipal Institute of Technology, Manipal Academy of Higher Education, Manipal, India, for supporting our research.

heading: Future Work
text: Future work should explore the impact of PEFT-similar models for lay and abstractive summarization. Limitations
Due to computational resource limitations, we were able to conduct only a limited number of experiments and were constrained by input token limits. Our access was limited to the Kaggle P100 GPU,
for which we are grateful. This restriction led us to primarily experiment with smaller model sizes, potentially missing the benefits of larger architectures. Our hyperparameter tuning was not extensive, and as a result, our models’ performance fell short of high scores on evaluation metrics. This indicates substantial room for improvement. We focused narrowly on sequence-to-sequence models, and these limitations present clear opportunities for future research to build upon and investigate