UIUC_BioNLP at BioLaySumm: An Extract-then-Summarize Approach Augmented with Wikipedia Knowledge for Biomedical Lay Summarization (2024)

Abstract: As the number of scientific publications is growing at a rapid pace, it is difficult for laypeople to keep track of and understand the latest scientific advances, especially in the biomedical domain. While the summarization of scientific publications has been widely studied, research on summarization targeting laypeople has remained scarce. In this study, considering the lengthy input of biomedical articles, we have developed a lay summarization system through an extract-then-summarize framework with large language models (LLMs) to summarize biomedical articles for laypeople. Using a fine-tuned GPT-3.5 model, our approach achieves the highest overall ranking and demonstrates the best relevance performance in the BioLaySumm 2024 shared task1.

text: Proceedings of the 23rd Workshop on Biomedical Language Processing, pages 132–143 August 16, 2024. ©2024 Association for Computational Linguistics
132

heading: 1 Introduction
text: New research in the biomedical field is often reported in the latest scientific articles and plays a crucial role in improving human health and wellbeing. However, the complex terminology and scientific language used in these publications can be challenging to understand for those without extensive knowledge of the field. The Biomedical Lay Summarization task (BioLaySumm) (Goldsack et al., 2024) addresses this challenge by creating summaries that are easier to read and understand, specifically tailored for readers unfamiliar with biomedical studies. Unlike traditional text summarization, which aims to condense documents into brief summaries, BioLaySumm also focuses on using easy-to-understand language. This approach ensures that the summaries are less technical and more accessible, while traditional summarization tasks prioritize capturing the precise scientific terminology used in the original documents. To solve
1Our code is available at https://github.com/ zhiwenyou103/UIUC_BioNLP_BioLaySumm2024.
the limited size and scope issue in current lay summarization corpora, Goldsack et al. (2022) have proposed two novel lay summarization datasets, PLOS and eLife, including biomedical journal articles alongside expert- and author-written lay summaries, which form the basis for the shared task.
We submit lay summaries generated by two finetuned LLMs for this task (one for each dataset). Considering the constraints on input size and computational resources (i.e., one GPU with 32 GB memory), using the full length of scientific articles for LLM fine-tuning is not feasible. Therefore, we adopt an extract-then-summarize approach (Koh et al., 2022; Bajaj et al., 2021) (illustrated in Figure 1), which allows us to reduce the input length while maintaining competitive performance. We fine-tune the Longformer EncoderDecoder (LED) model (Beltagy et al., 2020) and GPT-3.5 (OpenAI, 2024) and explore the effectiveness of combining unsupervised extractive summarization methods with a retrieval-augmented gener-
ation (RAG) approach (Guo et al., 2024) in generating lay summaries. Our experimental results on GPT-3.5 achieve the best overall ranking and the highest relevance score in the shared task.

heading: 2 Methods
text: We introduce our methodology for the biomedical lay summarization task (illustrated in Figure 1), including dataset description, section re-ranking, extractive summarization, RAG, GPT-3.5 fine-tuning, and evaluation measures. The detailed experimental settings are reported in Appendix A.

heading: 2.1 Datasets
text: We use eLife and PLOS datasets provided by Goldsack et al. (2022) for experiments. We report the average tokens of each dataset given the whole document and lay summarization in Appendix B. The lay summaries of eLife are crafted by expert editors, offering extensive abstraction and enhanced readability. Conversely, PLOS presents lay summaries written directly by the authors of articles. In terms of articles, eLife comprises peer-reviewed publications encompassing a broad spectrum of life sciences and medicine. PLOS covers journals spanning Biology, Computational Biology, Genetics, Pathogens, and Neglected Tropical Diseases (Goldsack et al., 2022).

heading: 2.2 Preprocessing
text: We employ two methods to preprocess the input article, aiming to reduce its length and extract salient sentences: section reordering and unsupervised extractive summarization.

heading: 2.2.1 Section Reordering
text: To better understand the experimental datasets, we conduct preliminary experiments to analyze which sections are most relevant to the gold standard lay summaries. We first group the headings of each article in eLife and PLOS datasets into five categories using structured section labels provided by the National Library of Medicine (NLM)2 (See Appendix C for more details). Then, based on the results of section-level similarity comparison, we reorder the whole article in the order of abstract, background, conclusion, results, and methods sections. The results of the reordering method in the
2https://lhncbc.nlm.nih.gov/ii/ areas/structured-abstracts/downloads/ Structured-Abstracts-Labels-102615.txt
eLife validation set in Appendix C show the effectiveness of the restructured article compared with the default section order.

heading: 2.2.2 Unsupervised Extractive Summarization
text: Given the input length constraints and limited computing resources, fully incorporating scientific articles for model fine-tuning is impractical. To capture the essential global information of the articles, we implement two unsupervised extractive summarization approaches: a graph-based ranking method (TextRank) (Mihalcea and Tarau, 2004) and a BERT-based clustering method (Miller, 2019), to extract salient sentences from the documents.
TextRank (Mihalcea and Tarau, 2004) operates by treating text as a graph, where nodes are constructed based on lemmas, parts-of-speech tags of tokens in the text, and edges based on cooccurrence within a window. By iteratively applying a ranking algorithm similar to Google’s PageRank (Brin and Page, 1998), it identifies the essential tokens, helping generate summaries or extract key information from text documents.
We use a BERT-based clustering approach (Miller, 2019) for unsupervised extractive summarization. It starts by dividing the article into segments using the LangChain3 NLTKTextSplitter API. Next, we apply a pre-trained embedding model PubMedBERT4 E deployed through SentenceTransformers5 to encode sentences from both lay summaries and segmented passages. We calculate the cosine similarity between these embeddings to create a contrastive learning dataset C, essential for fine-tuning the embedding model adapted for lay summarization tasks. Specifically, pairs with cosine similarity scores above 0.9 are considered positive, indicating high relevance, while those with scores below 0.01 are negative, indicating minimal relevance. We then fine-tune E with C through a contrastive loss. Appendix B presents the created dataset statistics of C. Following the method proposed by Miller (2019), we apply a K-means clustering approach to group sentences with the same themes and find the sentences closest to the cluster’s centroids as salient sentences. We extract the top 50 closest sentences from all clusters, each with a maximum length of 256
3https://www.langchain.com/ 4https://huggingface.co/NeuML/
pubmedbert-base-embeddings 5https://huggingface.co/sentence-transformers
tokens, as an extractive summary. These two extractive summarization approaches reduce the overall document length, capture the essential global context, and facilitate efficient model fine-tuning.

heading: 2.3 Retrieval-Augmented Generation
text: To simplify model-generated lay summaries, we resort to external knowledge due to the limited background information in the datasets. First, we use a keyword-based definition retrieval method to extract definitions from the Wikipedia dataset (Guo et al., 2024) through string matching. Specifically, we employ KeyBERT6 to extract the top 10 keywords from each article’s abstract using BERT embeddings. Then, we use dataset-provided and extracted keywords to retrieve short definitions from the Wikidata-based dataset (Guo et al., 2024). We use the Wikipedia API via LangChain for extended definitions if no results are found. We concatenate the retrieved information with the input article.
Additionally, we apply an embedding-based method to extract relevant information by selecting passages from the “wiki_dpr” dataset, which contains 21 million 100-word passages from Wikipedia (Lewis et al., 2020b). Using the pre-trained dense retrieval (DPR) component of the RAG model, we retrieve the five most relevant passages to integrate into our generation tasks.

heading: 2.4 GPT-3.5 Fine-Tuning
text: We also experiment with GPT-3.5-turbo7, a largescale closed-source model from OpenAI. Our experiments, along with findings from Turbitt et al. (2023), demonstrate performance below the baseline in zero-shot and few-shot prompting settings. Consequently, we investigate fine-tuning the model using the OpenAI API8. To minimize API costs, we employ the extract-then-summarize approach,
6https://github.com/MaartenGr/KeyBERT 7https://platform.openai.com/docs/models/
gpt-3-5-turbo 8https://openai.com/api/
utilizing TextRank to extract key sentences from the full text, which are then fed into the GPT model for summary generation. Our results indicate that fine-tuning on small random samples (100 to 400 examples) is adequate to achieve high performance for the task. For our final submission, we extract 40 sentences per article using TextRank and finetune separate models for each dataset using random samples of 200 articles.

heading: 2.5 Evaluation
text: We assess the performance of our model using the official evaluation scripts provided by the organizers (Goldsack et al., 2023), employing various automatic metrics related to relevance, readability, and factuality. Relevance is measured through ROUGE (1, 2, and L) (Lin, 2004) and BERTScore (Zhang et al.). Readability metrics include the FleschKincaid Grade Level (FKGL) (Kincaid et al., 1975), Dale-Chall Readability Score (DCRS) (Dale and Chall, 1948), Coleman-Liau Index (CLI) (Coleman and Liau, 1975), and LENS (Maddela et al., 2023). Notably, lower FKGL, DCRS, and CLI scores signify improved readability. Factuality evaluation incorporates AlignScore (Zha et al., 2023) and SummaC (Laban et al., 2022).

heading: 3 Results and Analysis
text: We present the evaluation results of our methods in the leaderboard in Table 1. Fine-tuning GPT3.5 ranks best overall among our submissions and outperforms the baseline BART model in all aspects. Additionally, we experiment with different numbers of sentences being extracted by the TextRank (Table 2) and different numbers of training examples on the eLife validation set (Table 3). We observe no significant improvement over various evaluation metrics when increasing the number of TextRank sentences beyond 40 and the training set size beyond 200 examples.
Despite GPT-3.5’s better performance over the smaller encoder-decoder models in most evaluation
aspects (Table 1), open-source models have some advantages, including reduced costs, the ability to fine-tune on various datasets, and reproducibility. Therefore, we conduct ablation studies on finetuning open-source models in Table 4. Initially, we compare two baseline configurations: the LED base model (LED-base4k) and the PubMed LED large model (PubMedLED4k), both using the top three sections as base input. We then apply the functional modules described in Section 2 to these baseline settings to assess the effectiveness of each component. In Table 4, DPR and Def refer to the RAG methods outlined in Section 2.3, which involve dense retrieval and entity-based definition retrieval from Wikipedia, respectively. Ext and TR denote the use of BERT-based unsupervised extractive summarization and TextRank, as introduced in Section 2.2.2. The term “All” represents the integration of all components, in the sequence of the top three sections, Ext, DPR, and Def, as input for finetuning. The results indicate the PubMed LED large
model achieves the highest LENS and relevance scores when all components are included. However, the readability scores do not surpass those of the LED base model.
Meanwhile, we notice that an article’s abstract achieves the highest factuality scores compared to the gold lay summary. Therefore, we explore the possibility of aligning the lay summaries generated by the PubMed LED model more closely with the article’s abstracts through GPT-4 post-processing. However, our experimental results indicate no apparent improvement across most evaluation metrics when using GPT-4 to enhance the alignment of the generated lay summaries with the abstracts (experimental details in Appendix E).

heading: 4 Discussion and Conclusion
text: Applying the extract-then-summarize framework to fine-tune GPT-3.5 demonstrates superior performance in the biomedical lay summarization task compared to LED-based fine-tuning. The ablation
study indicates that incorporating external knowledge during model fine-tuning slightly enhances relevance metrics in the experiments of the LEDbase model. However, it negatively impacts factuality scores, similar to the results observed when using extractive summarization and PubMed LED large model. None of the components enhance the factuality scores compared to the baseline settings, although there are improvements in relevance and readability scores (Table 4). We hypothesize that the external knowledge generated by RAG methods might contain noisy data, potentially affecting the factuality metrics. Additionally, the extractive summarizer may produce sentences with less contextual coherence than the original article, hindering the model’s ability to understand causal information during fine-tuning. While increasing the model size enhances relevance scores, it decreases readability and factuality from the LED base model to the PubMed LED large model.
The case study detailed in Appendix F reveals that the GPT-3.5 and PubMed LED models produce unrelated information when creating lay summaries compared to the gold lay summary. Notably, GPT3.5 produces longer summaries than the PubMed LED model despite both models having the same maximum decoding token limit of 512. Consequently, while GPT-3.5 includes more relevant sentences that closely match the original summary, it also introduces more irrelevant content.
Overall, fine-tuning GPT-3.5 with extractive summaries achieves the best overall ranking and highest relevance score in the BioLaySumm 2024 shared task, demonstrating the effectiveness of using key sentences from the article for LLM finetuning. The PubMed LED model, with additional functional components, also shows competitive results compared to GPT-3.5. Meanwhile, our findings using the PubMed LED model suggest a promising direction for future studies to develop advanced modules that combine extractive summarization and RAG to generate lay summaries, especially in improving the relevance scores and enhancing the accessibility of biomedical research.
Limitations
Our study’s limitations are as follows: 1) We conduct experiments using LED-based models for only one epoch with a small batch size due to time and computational constraints. We hypothesize that the model’s performance could vary with more tun-
ing epochs. 2) Our section reordering method may miss sections that do not match the NLM dictionary of section names, potentially impacting the model’s performance by omitting important content from articles. The proportions of mismatched sections are detailed in Appendix C. 3) The unsupervised extractive summarization methods used in this study are not tailored for lay summarization tasks, which may result in less relevant extraction. We suggest that developing a task-specific extractor could be a promising direction for future work. 4) We apply only two RAG methods in our experiments and concatenate the retrieved knowledge at the end of the input. The quality of the retrieved information was not filtered or verified, which may negatively impact fine-tuning performance. 5) Our method uses GPT-3.5 from OpenAI, which may not be fully reproducible since GPT-3.5 is a closed-source model and may update without unambiguous versions.

heading: Acknowledgement
text: This work is partially supported by the National Library of Medicine (NLM) of the National Institutes of Health under the award number R01LM014292.

heading: A Experimental Settings
text: We apply two baseline model types in our experiments: LED base9 (allenai/led-base-16384) and PubMed LED large10 (patrickvonplaten/led-large-16384-pubmed) models. Our final submission uses GPT-3.5 as the base model11.
Longformer Encoder-Decoder (LED). LED model is initialized from BART-base (Lewis et al.,
9https://huggingface.co/allenai/ led-base-16384
10https://huggingface.co/patrickvonplaten/ led-large-16384-pubmed
11https://platform.openai.com/docs/models/ gpt-3-5-turbo
2020a) as both models share the same architecture, with a maximum input length of 16,384 tokens.
PubMed LED large. PubMed LED large model is fine-tuned on the PubMed Summarization dataset (Cohan et al., 2018) through the checkpoint of led-large-16384.
GPT-3.5. GPT-3.5-turbo-1106 is fine-tuned on small random samples from the training datasets using the API provided by OpenAI.
We fine-tune the LED base and PubMed LED large models for 1 epoch and set batch size as 4. The maximum length of the decoder is 512. All experiments are conducted through one NVIDIA Tesla V100-32GB GPU. Considering memory efficiency, we use the default learning rate as 5e-5 for Adam optimization and set the floating point to 16 (i.e., fp16=True). For GPT-3.5 model fine-tuning12, we apply the default API hyper-parameters, along with default values (i.e., epochs=3, batch size=1, learning rate multiplier=8).
We compare different input text lengths using the LED-base model in Table 5. The results indicate that decreasing the input length affects the relevance scores. Specifically, LED16k, LED8k, and LED†4k show a consistent decrease in relevance scores, while other evaluation metrics exhibit fluctuations. Notably, LED16k achieves the lowest factuality scores compared to other settings, suggesting a need to reduce the length of the input article to help the model capture more factual information. The encoder maximum length we use for both
12https://platform.openai.com/docs/guides/ fine-tuning
eLife and PLOS datasets in Table 1 is 8,192 tokens due to the computing limitation.

heading: B Dataset Statistics
text: We report the average length of the article and lay summary in Table 6, as well as the statistics of two datasets in Table 7. As reported by Goldsack et al. (2022), there are no gold lay summaries for the test sets of both datasets for fair competition. In Table 8,
we show the contrastive learning datasets statistics for fine-tuning the embedding model introduced in Section 2.2.2.

heading: C Section Relevance
text: As introduced in Section 2.2.1, we apply a section re-ranking strategy in our experiments to deal with long input lengths. We first pair the headings that appear in each dataset with the structured abstract section labels provided by NLM, which contains 3,032 section labels and 5 corresponding broader NLM categories: BACKGROUND, OBJECTIVE, METHODS, RESULTS, and CONCLUSIONS. We report the heading matching proportions in Table 9. Specifically, for the eLife dataset, we identify 339 unmatched headings in the training set and 14 in the validation set, out of 21,315 and 1,158 headings, respectively. The PLOS dataset has 833 unmatched headings in the training set and 28 in the validation set, with overall totals of 122,873 and 6,800 headings, respectively. Notably, no OBJECTIVE sections are matched in either the eLife or PLOS datasets. In these cases, we concatenate the unmatched sections to the end of the article. Subsequently, we rank the sections by calculating the cosine similarity between each section’s content and the lay summary. We employ a pre-trained sentence transformer embedding
model, all-MiniLM-L6-v213, to encode both the lay summary and each section, allowing us to compute similarity scores. Figure 2 depicts the section relevance distribution in eLife and PLOS training sets. Our findings indicate that the most relevant sections for both the eLife and PLOS datasets are the “Abstract”, “Background”, and “Conclusion”, while the “Method” and mismatched “Other” sections are found to be less relevant.
Additionally, in Table 10, we compare the effectiveness of section reordering in an article by assessing the performance of models using ordered sections versus the original order. We use the LED-8k model on the eLife validation set for this evaluation. Specifically, we directly truncate the input as the natural order of the original dataset for the LEDoriginal model. Given our exploration of sections’ relevance with gold lay summary (Figure 2), we re-rank the sections based on the order of abstract-background-conclusion-resultsmethods. Therefore, we reorder the input of the LEDre-order model given the above order and truncate the model with an input limit of 8,192 tokens. Table 10 demonstrates that most evaluation scores improve with ordered sections as input, whereas ROUGE scores and DCRS show a decline.
13https://huggingface.co/sentence-transformers/ all-MiniLM-L6-v2

heading: D Prompt Template of GPT-3.5
text: We provide the prompt template for fine-tuning GPT-3.5 at the end of this section. The ### Article represents the output of extractive summarization using TextRank introduced in Section 2.2.2. The ### Summary denotes the gold lay summary of each article.
In the setting without fine-tuning (w/o FT), as introduced in Table 3, we use extractive summaries to prompt GPT-3.5 using the same template as finetuning GPT-3.5. The key difference in the w/o FT approach is that it does not include the goldstandard lay summary.
System: Generate a lay summary of this biomedical article
User: ### Article: Cell-fate reprogramming is at the heart of development... ### Summary:

heading: E Post-Processing using GPT-4
text: We observe the abstracts of articles achieve higher factuality scores compared to the gold lay summaries. As illustrated in Table 11, abstracts attain the highest factuality scores, while gold lay summaries achieve full relevance scores. We assess whether an LLM can reassemble model-generated lay summaries to resemble the articles’ abstracts more closely, thereby improving the factuality of the lay summaries. The input of GPT-4 includes a prompt template, original abstract, and PubMed LED model generated lay summary. While the AlignScore improves when using GPT-4 compared to the original settings, most other evaluation metrics, particularly readability scores, show a decline.
Due to the lack of factually accurate lay summaries as references to prompt or fine-tune GPT-4, the experiment is conducted solely under a zeroshot setting. We conclude that using zero-shot prompting with GPT-4 does not enhance the factuality of the generated lay summaries.
The prompt template used for GPT-4 is as follows:
System: You will be provided with a biomedical abstract and a corresponding lay summary.
Your task is to enhance the lay summary by integrating factual information from the abstract. Consider the abstract as an additional reference.
Make sure to keep the same wording as the provided lay summary but add more factual information. Do not change any factual information. Do not reduce the readability and relevance of your enhanced lay summary. Do not make up information.
Keep your enhanced lay summary roughly the same length as the provided lay summary.
User: Abstract: Cell-fate reprograming is at the heart of development... Lay Summary: Genes are the building blocks of life...

heading: F Case Study
text: To provide a more straightforward comparison of model-generated lay summaries, we conduct a case study comparing the generated lay summaries of GPT-3.5 and the PubMed LED large models in the
test sets of eLife and PLOS in Table 12 and Table 13. We randomly select an article with ID elife78005-v1 from eLife and journal.pgen.1008255 from PLOS. Our findings reveal that both the finetuned GPT-3.5 and the PubMed LED model generate irrelevant information not mentioned in the gold lay summary in both datasets. However, GPT3.5 produces more informative sentences than the PubMed LED large model.

heading: Future Work
text: Meanwhile, our findings using the PubMed LED model suggest a promising direction for future studies to develop advanced modules that combine extractive summarization and RAG to generate lay summaries, especially in improving the relevance scores and enhancing the accessibility of biomedical research. Limitations
Our study’s limitations are as follows: 1) We conduct experiments using LED-based models for only one epoch with a small batch size due to time and computational constraints. We hypothesize that the model’s performance could vary with more tun-
ing epochs. 2) Our section reordering method may miss sections that do not match the NLM dictionary of section names, potentially impacting the model’s performance by omitting important content from articles. The proportions of mismatched sections are detailed in Appendix C. 3) The unsupervised extractive summarization methods used in this study are not tailored for lay summarization tasks, which may result in less relevant extraction. We suggest that developing a task-specific extractor could be a promising direction for future work. 4) We apply only two RAG methods in our experiments and concatenate the retrieved knowledge at the end of the input. The quality of the retrieved information was not filtered or verified, which may negatively impact fine-tuning performance. 5) Our method uses GPT-3. 5 from OpenAI, which may not be fully reproducible since GPT-3. 5 is a closed-source model and may update without unambiguous versions