RAG-RLRC-LaySum at BioLaySumm: Integrating Retrieval-Augmented Generation and Readability Control for Layman Summarization of Biomedical Texts (2024)

Abstract: This paper introduces the RAG-RLRCLaySum framework, designed to make complex biomedical research understandable to laymen through advanced Natural Language Processing (NLP) techniques. Our Retrieval Augmented Generation (RAG) solution, enhanced by a reranking method, utilizes multiple knowledge sources to ensure the precision and pertinence of lay summaries. Additionally, our Reinforcement Learning for Readability Control (RLRC) strategy improves readability, making scientific content comprehensible to non-specialists. Evaluations using the publicly accessible PLOS and eLife datasets show that our methods surpass Plain Gemini model, demonstrating a 20% increase in readability scores, a 15% improvement in ROUGE-2 relevance scores, and a 10% enhancement in factual accuracy. The RAG-RLRC-LaySum framework effectively democratizes scientific knowledge, enhancing public engagement with biomedical discoveries 1.

text: Proceedings of the 23rd Workshop on Biomedical Language Processing, pages 810–817 August 16, 2024. ©2024 Association for Computational Linguistics
810

heading: 1 Introduction
text: Biomedical research encompasses crucial discoveries, ranging from everyday health concerns to significant disease outbreaks. Such studies are essential not only for scientists and doctors but also for journalists and the general public. However, the specialized and complex language typical in these studies often renders the content incomprehensible to those without a scientific background Thoppilan et al. (2022). To address this issue, the development of automated lay summaries have become increasingly important Goldsack et al. (2023b,a). This initiative aims to summarize the detailed aspects of biomedical research into summaries that are both comprehensible and devoid of complicated
1Our code and implementation details are available here: https://github.com/JoyDajunSpaceCraft/RAG-RLRCLaySum
jargon. Although these systems show great potential, doubts about their accuracy are a major obstacle to their widespread useGabriel et al. (2020); Maynez et al. (2020); Yang et al. (2024); Li et al. (2023b, 2024); Wang et al. (2024). Our framework integrates specific external explanations for complex terms to further enhance content simplification. In response to the concerns about the integrity of summarized information, our framework employs a “knowledge retrieval” approach within the Retrieval-Augmented Generation (RAG) framework. This method uses a neural re-ranker to dynamically integrate trustworthy external knowledge sources like Wikipedia, ensuring that summaries are simplified, factually accurate, and contextually relevantLewis et al. (2020); Kang et al. (2024).
The architecture of the proposed RAG is illustrated in Figure 1. We also introduce a rewardbased approach to overcome the limitations of traditional fine-tuning, which often produces summaries that have high ROUGE scores but are not actually readable to humans. This method fine-tunes the model by rewarding outputs that align with read-
ability metrics (Flesch-Kincaid Grade Level and Dale-Chall Readability Score Foster and Rhoney (2002); Ribeiro et al. (2023)). Unlike traditional supervised methods that might limit the model‘s adaptability, our approach encourages the model to alternative expressions to enhance clarity. The BioLaySumm challenge is a research competition that focused on developing and benchmarking models for generating lay summaries from complex biomedical literature2. Our method is ranked 11th on the leaderboard of the challenge Goldsack et al. (2024).

heading: 2 Methodology
text: First, we employ a Retrieval-Augmented Generation (RAG) solution that processes entire papers despite limited input capacity. Second, we improve summary quality by optimizing readability using relevant background information. The framework is illustrated in Figure 2.

heading: 2.1 Retrieval Augmented Generation (RAG)
text: Our RAG framework enhances keyword-based retrieval by using an initial lay summary generated
2https://biolaysumm.org/
by the model as a query. In the inference stage, it retrieves relevant descriptions from Wikipedia Ponzetto and Strube (2007) by the Pyserini Lin et al. (2021) index. However, retrieving relevant information from a large number of articles remains a challenge because the first generated summaries cannot work as effective queries. We initially use the ground truth as a query but switch to the first generated layman summary during inference for passage retrieval. However, there’s a risk that the top-k passages may not be the most relevant for generating accurate summaries. Given a scientific document D with a set of candidate passages K = {k1, k2, ..., kn} from grounding sources, the RAG framework generates a lay summary S by maximizing the probability:
p(S|D,K) = |S|∏ i=1 p(si|s<i, D,K) (1)
where si represents the i-th token in the summary, and s<i denotes the sequence of tokens preceding si. We use ColBERT Khattab and Zaharia (2020) and BGE-v2 Li et al. (2023a); Chen et al. (2024) as two different types of the neural re-ranker. The details about the trained re-ranker are in Ap-
pendix B.

heading: 2.2 Reinforcement Learning for Readability Control (RLRC)
text: For details about the reranking model and sequence generation model training can be seen in Appendix A. The RLRC method inputs the first generation from the plain LED and uses the ground truth as the expected output. Our RLRC approach employs a reinforcement learning strategy to fine-tune the readability of summaries. We define a reward function R(y, r∗) based on the desired readability level r∗ that encourages the generation of text towards better readability, which is measured by the Flesch Reading Ease score r∗:
R(y, r∗) = 1− exp ( − (R(y)− r ∗)2
2σ2
) (2)
where R(y) denotes the readability score of the generated summary y, and σ is a hyperparameter that controls the sensitivity of the reward function to deviations from the target readability score. Also, we leverage a Gaussian-based reward that strongly penalizes great variations in the readability Ribeiro et al. (2023).
We employ the Proximal Policy Optimization (PPO) Schulman et al. (2017) algorithm to optimize our RLRC model. The objective is to adjust the model’s parameters θ by maximizing the objective function:
L(θ) = E(y,r∗)∼pθold [( pθ(y | D, r∗) pθold(y | D, r∗) ) R(y, r∗) ] (3)
Here, pθoldand pθ denote the policy under the old and current parameters, respectively.

heading: 2.3 Large Language Models
text: We use the LLMs in two ways: first, as a paraphrasing tool during inference to refine initial generations, and second, for directly generating layman summaries. This implementation is built on Gemini-1.0-pro, developed by Google Team et al. (2023), which also serves as our baseline LLM. We aim to create readable summaries while incorporating as many input keywords as possible. We follow Gemini-1.0-pro’s default settings, and the prompt details are described in Appendix C.

heading: 3 Experimental Settings and Results

heading: 3.1 Datasets and Evaluation
text: This study uses biomedical research articles from the PLOS and eLife datasets, which include both
technical abstracts and expert-crafted lay summaries. The PLOS dataset contains 24,773 training and 1,376 validation instances, while the eLife dataset comprises 4,346 training and 241 validation instances Goldsack et al. (2022). We assess summarization quality using predefined metrics: Relevance is gauged by ROUGE Scores (ROUGE-1, ROUGE-2, ROUGE-L) Lin (2004) and BERTScore Zhang et al. (2019); Readability by the Flesch-Kincaid Grade Level (FKGL) Kincaid et al. (1975), Dale-Chall Readability Score (DCRS) Dale (1948), and Learnable Evaluation Metric for Text Simplification (LENS) Maddela et al. (2022); Factuality by Summac Laban et al. (2022) and AlignScore Zha et al. (2023).

heading: 3.2 Performance of Baseline Models
text: The Plain LED model, serving as our baseline, achieved ROUGE-L scores of 41.33 and 44.07. In contrast, the Plain retrieve+LED model, which integrates external knowledge through the BM25 retriever, slightly improved ROUGE-L scores to 47.02 and 47.21. This indicates that the incorporation of external knowledge slightly enhances the relevance of the summaries.

heading: 3.3 Effect of Neural Re-rankers
text: Further improvements were observed with the RAG+LED model, which incorporates a trained neural re-ranker, boosting the ROUGE-L scores to 49.68 and 49.79. This significant increase demonstrates that neural re-rankers are more precise in selecting relevant content, effectively enhancing the accuracy and relevance of the summaries.

heading: 3.4 Effect of Large Language Models
text: The RAG+ChatGPT and RAG+Gemini models, utilizing LLMs, achieved high FKGL readability scores of 9.93 and 9.25 respectively, but their ROUGE-L scores were lower at 39.59 and 39.20, indicating that LLMs can sometimes introduce irrelevant information. Similarly, the Plain Gemini model, which relies solely on an LLM, scored only 33.63 in ROUGE-L, demonstrating the challenges LLMs face in producing coherent and accurate summaries without mechanisms for precise content selection.

heading: 3.5 Effect of RLRC
text: The RAG+RLRC model, integrating reinforcement learning training strategies, achieved a ROUGEL score of 47.24. It marked an improvement in
factual accuracy, with a Summac score of 78.45 compared to the 73.44 of Plain LED. This highlights the effectiveness of reinforcement learning strategies in optimizing the text’s factual alignment.

heading: 4 Related Work
text: Automatic summarization in the biomedical domain has been extensively studied Du et al. (2019); Krishna et al. (2020); Goldsack et al. (2023a); Devaraj et al. (2022). The primary challenge in this field is simplifying the content of original articles to make them comprehensible to laypersons. While Rosati (2023) supplement source documents to aid in generating more comprehensible summaries, and Devaraj et al. (2022) explore how text simplification impacts summary accuracy, introducing a taxonomy of error types and identifying omissions as a prevalent issue, these approaches often overlook the balance between simplification and factuality.
To enhance summary factuality, researchers incorporate factual knowledge from external sources during model training Mao et al. (2022), which has proven effective in improving accuracy. Rosati (2023) utilize Wikipedia to enrich summaries with
additional knowledge, while Poornash et al. (2023) employ a trained re-ranker to select pertinent information, enhancing the factuality of summaries.

heading: 5 Conclusion and Future Work
text: The RAG-RLRC-LaySum framework effectively simplifies complex biomedical texts, enhancing readability and factual accuracy for lay audiences. It surpasses traditional models, offering new insights into the pivotal role of knowledge retrieval and readability optimization in scientific summarization. Future work will expand the framework’s knowledge sources and refine how knowledge is utilized, potentially broadening its application across various scientific fields. This will further explore the integration of domain-specific knowledge to improve the precision and relevance of summaries.

heading: 6 Limitations
text: While the RAG-RLRC-LaySum framework shows promise, it has several limitations. The reliance on external sources like Wikipedia can introduce biases. The framework’s computational complexity is high, making real-time applications challeng-
ing. Readability metrics like FKGL and DCRS may not fully capture readability for all audiences. Additionally, the generalizability to other domains beyond biomedical texts is uncertain. Lastly, evaluations based on automated metrics may not fully reflect user experience, highlighting the need for human evaluations. Future work should address these limitations by exploring diverse knowledge sources, optimizing efficiency, refining readability metrics, and conducting human evaluations.

heading: A Finetuning models
text: For training the Longformer Encoder-Decoder (LED) model Beltagy et al. (2020), we utilized the "allenai/led-base-16384" pre-trained checkpoint available on Huggingface’s model hub. Our training setup included a configuration that processes 16,384 input tokens and generates outputs limited to 512 tokens. This training was conducted over the course of a single epoch.
In parallel, we employed BioLinkBERT-base Yasunaga et al. (2022) as the foundational language model for processing eLife and PLOS datasets, leveraging its specialized capabilities in understanding biomedical context.
Then, we designed a neural re-ranker based on the ColBERT Khattab and Zaharia (2020) and BGE-v2 Li et al. (2023a); Chen et al. (2024) scoring mechanism, which refines the results by evaluating the relevance of retrieved documents. The training for this re-ranker was tailored to accept inputs of up to 512 tokens, and it was fine-tuned to generate models by considering the top 5 most pertinent retrieval results. Futhermore, we define the Flan-T5-Large from huggingface, we use the model "google/flan-t5-large" as the base model. To make use of the control generation, we use the keywords in the article as the expected output to make sure the relevance.
A.1 Retrieval Augmented Generation We conduct the experiment based on the model Longformer Encoder-Decoder (LED) Beltagy et al. (2020) which supports an input token length of 16,384 tokens. For the basic fine-tuning method, we find out in both the PLOS and eLife data that the re-ranker result will be a higher result in the RougeL and a lower score in the FKGL and DCRS score. In that case, indicate the lower the complexity of the description.
We use ColBERT Khattab and Zaharia (2020) and BGE-v2 Li et al. (2023a); Chen et al. (2024) as two different types of the neural re-ranker.
A.2 Reinforcement Learning for Readability Control (RLRC)
By utilizing various control levels for readability within the model-generated results, we focus on understanding how modifications to the readability scores, particularly the Flesch-Kincaid Grade Level (FKGL), impact the final summaries. The Flan-T5 model Chung et al. (2024) serves as the primary backbone for text generation. During the inference phase on testing data, where no ground truth is available for the reward mechanism, keywords are used as proxy indicators to ensure that the generated summaries accurately reflect the expected concepts.
In our model, we define two key mathematical expressions. The first is the Gaussian probability density function, used to estimate the likelihood of a given value within a normal distribution. The expression for this function is:
calc_nd(value,mean) =
1
σ √ 2π exp
( −(value−mean) 2
2σ2 ) (4) This function is essential for assessing how far a data point deviates from the mean and is widely used in statistical analyses.
The second formula defines our reward function, which combines three different scoring metrics—readability score, BERTScore, and text length score—to comprehensively evaluate the quality of the text. The formula is as follows:
reward = wr · normalized_flesch_scores+ wb · all_bertscore_scores + wl · length_scores (5)
Here, wr, wb and wl are the weight factors for each scoring metric, adjusting the influence of each
score in the overall assessment. By default, we set wr = 0.5, wb = 0.3, wl = 0.2.
This weighted approach allows us to tailor the scoring criteria to different types of text analysis tasks, accommodating the multifaceted nature of text data.

heading: B Retrieval Design
text: For the reranking of retrieved documents, we utilize the pyserini package Lin et al. (2021). Following the approach outlined by Rosati (2023), we employ enwiki−paragraphs for background knowledge. We first retrieved 20 candidate paragraphs and then rerank the top 5 results.
B.1 Neural Re-ranker In the provided Table 2, the performance trends across the eLife and PLOS datasets reveal that neural re-ranking methods (ColBERT and BGE) consistently outperform the traditional BM25 method. Notably, BGE shows a clear upward trend in accuracy from Top1 through Top20 in both datasets. Similarly, ColBERT’s performance also exhibits an upward trajectory, although it remains below BGE, indicating a strong but second-tier efficacy among the tested methods.

heading: Future Work
text: Future work should address these limitations by exploring diverse knowledge sources, optimizing efficiency, refining readability metrics, and conducting human evaluations