Pathology Dynamics at BioLaySumm: the trade-off between Readability, Relevance, and Factuality in Lay Summarization
Year: 2023

Abstract:
Lay summarization aims to simplify complex scientific information for non-expert audiences. This paper investigates the trade-off between readability and relevance in the lay summarization of long biomedical documents. We introduce a two-stage framework that attains the best readability metrics in the first subtask of BioLaySumm 2023, with 8.924 Flesch–Kincaid Grade Level and 9.188 Dale–Chall Readability Score. However, this comes at the cost of reduced relevance and factuality, emphasizing the inherent challenges of balancing readability and content preservation in lay summarization. The first stage generates summaries using a large language model, such as BART with LSG attention. The second stage uses a zero-shot sentence simplification method to improve the readability of the summaries. In the second subtask, a hybrid dataset is employed to train a model capable of generating both lay summaries and abstracts. This approach achieves the best readability score and shares the top overall rank with other leading methods. Our study underscores the importance of developing effective methods for creating accessible lay summaries while maintaining information integrity. Future work will integrate simplification and summary generation within a joint optimization framework that generates highquality lay summaries that effectively communicate scientific content to a broader audience. Code: https://github.com/iah3/ readability-summarization

text: The 22nd Workshop on Biomedical Natural Language Processing and BioNLP Shared Tasks, pages 592–601 July 13, 2023 ©2023 Association for Computational Linguistics

heading: 1 Introduction
text: The burgeoning volume of biomedical literature in recent years has posed significant challenges for researchers, healthcare professionals, and the general public in staying abreast of the wealth of information generated. The task of manually summarizing long-form documents has become increasingly impractical, requiring a disproportionate amount of effort and domain-specific knowledge (Alomari
et al., 2022; Adams et al., 2023; Phang et al., 2022; Al-Hussaini et al., 2022; Zhang et al., 2022a). Automatic text summarization, which seeks to distill source texts while retaining their core ideas, continues to be a demanding task, especially with long, content-rich documents laden with domainspecific complexities (Guo et al., 2022; Cao and Wang, 2022; Zhang et al., 2022b; Mao et al., 2022; Manakul and Gales, 2021).
As such, there is an urgent need to develop effective summarization techniques tailored for extensive biomedical documents to cater to diverse audiences (Goldsack et al., 2022; Ondov et al., 2022; Moro et al., 2022; Bishop et al., 2022). Meanwhile, computational complexity persists as a major obstacle, with specialized hardware potentially paving the way for more energy-efficient implementations (Gong et al., 2022; Hah et al., 2022; Athena et al., 2022a,b; West et al., 2023). As we continue to tackle these challenges, the field is poised for advancements that will fundamentally reshape how we interact with and benefit from the biomedical literature.
Lay summarization simplifies and distills complex scientific information into an accessible format for non-experts (Goldsack et al., 2023, 2022). It is vital for bridging the gap between specialized knowledge and the broader community. Controlled summarization can further enhance the accessibility of biomedical research findings by ensuring that generated summaries are both informative and comprehensible through readability. Readabilitycontrolled summarization can maximize the use of scientific knowledge and allow various stakeholders to make informed decisions in healthcare and research (Luo et al., 2022). Generating lay summaries for long documents poses unique challenges due to the inherent complexity of the subject matter and the specialized language used in the original documents. Balancing the simplification of language with the preservation of accurate and
592
relevant information is crucial. Reducing jargon and technical terminology may lead to the loss of essential details or the introduction of errors.
The BioNLP 2023 Workshop at ACL recently introduced a new shared task BioLaySumm, focusing on lay summarization of biomedical research articles (Goldsack et al., 2023). It comprises two subtasks with distinct objectives. The first subtask aims to generate lay summaries from PLOS and eLife articles, including the abstract (Goldsack et al., 2022), striving to maximize relevance and factuality metrics while minimizing readability score metrics. The second subtask focuses on readability-controlled summarization, which seeks to maximize relevance and factuality scores and generate both abstracts and lay summaries with readability levels comparable to the target summaries.
This paper investigates techniques for generating lay summaries and readability-controlled abstracts for long biomedical documents. We focus on their effectiveness in maintaining relevance, factuality, and readability. The ultimate goal is to facilitate knowledge dissemination and empower diverse audiences to engage with complex scientific information (Goldsack et al., 2023). A multistep approach involving Bidirectional and AutoRegressive Transformers (BART) (Lewis et al., 2020) and Multilingual Unsupervised Sentence Simplification (MUSS) (Martin et al., 2022) obtains the highest readability scores in the first subtask of generating lay summaries at the cost of lower relevance and factuality. In the second subtask, an approach based on Local, Sparse and Global (LSG) attention (Condevaux and Harispe, 2022) obtains the highest readability and joint highest overall scores.

heading: 2 Related Work
text: In recent years, biomedical document summarization has benefitted from advancements in deep learning and language models (Zhang et al., 2019a; Wang et al., 2021). Wallace et al. (Wallace et al., 2021) investigated BART (Lewis et al., 2020) for summarization of randomized controlled trials. Sotudeh et al. (Sotudeh Gharebagh et al., 2020) improved radiology report summarization by incorporating medical ontology into a sequence-tosequence model. Domain-specific corpora using abstracts as summaries (Cohan et al., 2018; Wang et al., 2020b), has also contributed to the field.
DeYoung et al. (DeYoung et al., 2021) examined the summarization of systematic reviews based on cited clinical trials. In contrast, Guo et al. (Guo et al., 2021) combined summarization and simplification to generate plain language summaries from abstracts of systematic reviews.

heading: 2.1 Lay Summarization
text: Prior lay summarization work primarily originates from the Shared Tasks at Scholarly Document Processing 2020: LaySumm (Chandrasekaran et al., 2020). AUTH (Gidiotis et al., 2020) employs a PEGASUS-based method to compress and rewrite article abstracts, fine-tuning the model to generate lay summaries. Dimsum (Yu et al., 2020) generates summaries using a joint extractive and abstractive approach, leveraging BART encoder and training with both extractive and abstractive summarization objectives. Kim (Kim, 2020) primarily utilizes the PEGASUS model, combining it with a BERTbased extractive model and incorporating readability metrics to enhance summary quality. Reddy et al. (Reddy et al., 2020) adopt an unsupervised extractive sentence classification method using variants of the maximum marginal relevance metric. Summaformers (Ghosh Roy et al., 2020) leverages the BART model, trained on the CNN/Dailymail dataset and fine-tuned on the LaySumm corpus. Mishra et al. (Mishra et al., 2020) employs a standard encoder-decoder framework for abstractive summarization based on BERT fine-tuned on the CNN/Dailymail dataset. Chaturvedi et al. (Chaturvedi et al., 2020) uses a two-stage pipeline involving extractive summarization, sentence extraction, and BART model-based summarization of selected text segments. However, these works were only evaluated on ROUGE score (Chaganty et al., 2018; Kryscinski et al., 2019). The recent study by Goldsack et al. (Goldsack et al., 2022) revealed that employing extractive methods or merely utilizing the abstract can lead to elevated ROUGE scores while sacrificing readability. Furthermore, the research demonstrated the capacity of a BARTbased model (Lewis et al., 2020) to generate lay summaries with both high relevance scores and low readability scores, thus achieving the desired outcome.

heading: 2.2 Readability, Relevance, and Factuality
text: Readability, which reflects the ease of understanding a text, is influenced by factors like lexical and syntactic complexity, discourse cohesion, and back-
ground knowledge (Crossley et al., 2017). In this study, we evaluate readability using Flesch-Kincaid Grade Level (FKGL) (Kincaid et al., 1975; Flesch, 2007) previously employed in lay summarization research (Guo et al., 2021), and the Dale-Chall Readability Score (DCRS) (Dale and Chall, 1948; Chall and Dale, 1995).
Relevance metrics like ROUGE (Chaganty et al., 2018; Kryscinski et al., 2019) and BERTScore (Zhang et al., 2019b) assess if a summary captures the source’s main ideas. Factual consistency metrics evaluate summary-source consistency (Goyal and Durrett, 2020; Wang et al., 2020a). Despite high factual error rates in short-document modelgenerated summaries (Cao et al., 2018; Maynez et al., 2020), efforts have focused on developing effective factuality metrics (Honovich et al., 2021; Xie et al., 2021; Ribeiro et al., 2022). ROUGE scores best correlate with human relevance scores, providing a platform for benchmarking long document abstractive models (Koh et al., 2022). According to Koh et al., the best overall correlation with human factual consistency scores is achieved by fine-tuned BARTScore (Koh et al., 2022).

heading: 3 Data
text: The shared task leverages data from two principal sources, the Public Library of Science (PLOS) (Goldsack et al., 2022, 2023; Luo et al., 2022) and eLife (Goldsack et al., 2022, 2023). Each of these datasets comprises biomedical research articles, paired with their technical abstracts and lay summaries written by experts. As discussed in the preceding section, the utility of each type of summary differs depending on the subtask. Moreover, the lay summaries in each dataset exhibit several distinct characteristics; for more comprehensive details, readers are referred to (Goldsack et al., 2022, 2023; Luo et al., 2022).
The PLOS dataset, the larger of the two, contains 24,773 instances designated for training and 1,376 instances for validation, applicable to both subtasks. In contrast, the eLife dataset consists of 4,346 training instances and 241 validation instances.
The test data for subtask 1 includes 142 articles each from PLOS and eLife. Meanwhile, the test data for subtask 2 comprises 142 PLOS articles, distinct from those used in subtask 1.
It’s important to note that the test sets allocated for both subtasks are distinct from those published
in any of the cited papers, and are made accessible via the CodaLab pages dedicated to the shared task. This configuration fosters a rich and diverse dataset, providing comprehensive support for the objectives of the shared task.

heading: 3.1 Metrics
text: In order to effectively gauge the performance of the proposed models, a range of comprehensive metrics were assigned in the evaluation.
To measure the relevance of the summaries to their source documents, ROUGE-1, ROUGE-2, and ROUGE-L metrics (Chaganty et al., 2018; Kryscinski et al., 2019) were assigned. These metrics provide insights into summary quality by comparing them with human-authored references, specifically by calculating the overlap of n-grams between the generated summary and the source text. ROUGE-1, ROUGE-2, and ROUGE-L evaluate the overlap of unigrams, bigrams, and longest common sequences, respectively.
BERTScore (Zhang et al., 2019b) was also assigned for relevance evaluation. BERTScore uses BERT embeddings to calculate semantic similarity between generated summaries and source texts, providing a more refined evaluation than raw n-gram overlap.
To evaluate readability, Flesch-Kincaid Grade Level (FKGL) (Kincaid et al., 1975; Flesch, 2007) and Dale-Chall Readability Score (DCRS) (Dale and Chall, 1948; Chall and Dale, 1995) were used. These metrics measure text understanding ease, considering elements such as syntactic complexity, lexical diversity, and sentence length. FKGL estimates the US school grade level required to understand the text, while DCRS measures complexity based on word familiarity and sentence length.
The factual consistency of the produced summaries is an essential aspect in summarization tasks. For this purpose, BARTScore (Koh et al., 2022) was employed. This metric measures the consistency between summary and source, pinpointing factual inconsistencies and ensuring that the summaries accurately reflect the original source content. As per the findings of Koh et al., BARTScore has the best overall correlation with human factual consistency scores, making it a reliable choice for this evaluation.
The application of these metrics in the evaluation process facilitates a comprehensive analysis of the generated summaries’ quality, relevance, read-
Table 1: Lay summarization scores on the test set on CodaLab. The proposed methods (PD 1, PD2, PD3) are delineated from the baseline and the top-ranked team, MDC, by a line. Lower scores are favorable for FKGL and DCRS, while higher scores are advantageous for the other metrics, as illustrated by the arrows.
ROUGE-1 ↑ ROUGE-2 ↑ ROUGE-L ↑ BERTScore ↑ FKGL ↓ DCRS ↓ BARTScore ↑ Baseline 0.470 0.145 0.437 0.864 12.069 10.249 -0.831 MDC 0.482 0.155 0.449 0.871 12.937 10.206 -1.177
PD 1 - BART1 0.494 0.159 0.460 0.859 13.096 10.122 -2.331 PD 2 - BART1 + MUSS 0.475 0.135 0.448 0.854 8.924 9.188 -3.230 PD 3 - LSG2 0.473 0.148 0.438 0.857 12.488 9.986 -2.178
Figure 1: Sentence Simplification in Lay Summary Generation for Subtask 1 using MUSS (Martin et al., 2022). Although readability improves, relevance and factuality suffers due to the inclusion of an abbreviation without clarification.
ability, and factual accuracy.

heading: 4 Lay Summarization
text: The generation of lay summaries was executed through a two-stage method. Prior to delving into this method, it’s important to outline the baseline approach from which it sprang. This involved using a base BART model (Lewis et al., 2020), pretrained on the CNN Daily Mail (Hermann et al., 2015; See et al., 2017), with an input and output token length of 10241. Any articles exceeding 1024 tokens in length were truncated. This approach was
1facebook/bart-large-cnn 2ccdv/lsg-bart-base-4096 3 mrm8488/t5-base-finetuned-summarize-news
the most successful of the three proposed methods when generating abstractive lay summaries, achieving the highest overall scores, primarily due to its superior relevance score, which is consistent with the findings of Goldsack et al. (Goldsack et al., 2022). This approach is referred to as PD 1 (Pathology Dynamics) in Tables 1 and 2.
However, this BART-based approach fell short in terms of readability scores. To remedy this, the twostage method was implemented. In the first stage, a large language model such as BART (Lewis et al., 2020) or LSG (Condevaux and Harispe, 2022) was trained to generate lay summaries based on the articles, which incorporated abstracts, containing the most important information in a concise format. This eliminated the need for longer input token lengths required in Subtask 2, which did not include the abstract in the input.
In the second stage, the two-stage approach used a zero-shot sentence simplification method called MUSS (Martin et al., 2022) to enhance the readability of the generated lay summaries. This approach, referred to as PD 2 in Tables 1 and 2, achieved the best readability score among all the participating teams in the subtask, attesting to the effectiveness of integrating MUSS (Martin et al., 2022) into the process.
Despite this success, Figure 1 indicates certain limitations with MUSS (Martin et al., 2022). While readability was improved, certain aspects of clarity were compromised, such as the detachment of the abbreviation OPC from its full form, oropharyngeal candidiasis. As a result, the summary’s relevance and factuality scores dipped below that of the base model, emphasizing the need for a careful balance between simplification techniques and the maintenance of essential information.
The final method involved the use of an LSG attention model (Condevaux and Harispe, 2022) with an increased input token length of 4096 for the encoder to generate lay summaries, while keeping
Table 3: Readability-controlled summarization scores on the test set on CodaLab. The proposed method is distinguished from other approaches by a line. Lower scores are preferable for FKGL and DCRS, while higher scores are desired for the remaining metrics, as denoted by the arrows.
Team ROUGE-1 ↑ ROUGE-2 ↑ ROUGE-L ↑ BERTScore ↑ FKGL ↓ DCRS ↓ BARTScore ↑ Baseline 0.409 0.116 0.369 0.855 2.396 0.931 -0.978 NCUEE-NLP 0.451 0.140 0.412 0.855 2.048 0.934 -2.110 LHS712EE 0.442 0.130 0.405 0.855 2.263 0.936 -1.140
Pathology Dynamics 0.451 0.138 0.410 0.853 2.107 0.823 -1.568
Figure 2: Generation of abstracts and lay summaries for Subtask 2. The figures on the left depict the results from the two-stage approach. Text boxes on the right display the original abstract of the article and the desired lay summary. Upon manual review and metric evaluation using the validation set, it is noted that the output from stage 2 diverges from the target lay summary. As indicated by the arrows, the output from stage 1 therefore acts as both the lay summary and the abstract for the purposes of evaluation.
the decoder’s output token length at 10242. This approach, denoted as PD 3 in Table 1, yielded the most balanced results and achieved the highest factuality among the proposed methods, as seen in Table 2. Thanks to its efficient attention mechanism, the LSG model’s training was as swift as BART’s, despite a longer input token length.

heading: 5 Readability Controlled Summarization
text: This subtask was designed to yield lay summaries and abstracts from other sections of the article using a singular model. In order to address this intricate task, we initiated our methodology with a pre-processing phase designed to create a synthetic dataset. This was achieved by duplicating each article and correlating it with the corresponding abstract and lay summary as outputs. As a result, this tactic effectively doubled the number of articles available for training in our dataset, representing a
significant enhancement over the volume offered by the original dataset.
Following the augmentation of our dataset, we utilized this expanded synthetic dataset to train a model based on LSG (Condevaux and Harispe, 2022). This particular model was adept at generating a summary that seamlessly integrated components from both the lay summary and the article. One notable characteristic of this subtask was the absence of an abstract in the input articles. This invariably meant that crucial information was scattered throughout the entirety of the article, thereby calling for a greater input token length.
To account for this, we engaged an LSG-based model (Condevaux and Harispe, 2022), specifically configured with an encoder that accommodated 4096 input tokens and a decoder that handled 1024 output tokens2. This specialized setup surpassed the performance metrics exhibited by T53 (Raffel
et al., 2020) and BART1 (Lewis et al., 2020) models with 1024 input tokens , as per the results from the validation set. Consequently, this led to the selection of the LSG-based model for the following stages of the task, underlining its capability to deliver effective summarization for long biomedical documents. The results are presented in Tables 3 and 4.
Moving forward, we incorporated the two-stage modeling approach, as described in the previous task, into this current subtask. In the initial stage, the LSG-based model (Condevaux and Harispe, 2022) was tasked with generating a summary intended to act as the abstract. The subsequent stage hinged on the use of MUSS (Martin et al., 2022) to produce the lay summary. However, an in-depth inspection of the generated summaries coupled with an analysis of the validation set scores led us to the realization that the output from the first stage of the model served as a more effective lay summary than that produced by MUSS. We found that the MUSS output was excessively simplified when juxtaposed against the target lay summary.
Due to this observation, we opted for the utilization of the output from the first stage to fulfill the dual roles of both the abstract and the lay summary. The objective of this task was to achieve readability scores that more closely aligned with the target values, as opposed to pursuing the lowest possible scores. Given this context, the application of MUSS resulted in a decline in readability scores for this subtask.
Figure 2 presents the opening sentences of the outputs from the first and second stages for a representative article. This visual comparison highlights the propensity of MUSS (Martin et al., 2022) to oversimplify the summary, causing it to diverge from the intended lay summary. As a result, the utilization of the same output from the first stage for both the abstract and lay summary culminated in superior scores, even in terms of readability. This is because the readability score for this subtask involved a comparison between the generated lay
summary and the original version. This points to the inherent challenge of striking a balance between readability and the retention of crucial content when employing simplification strategies for the generation of lay summaries.

heading: 6 Conclusion and Future Work
text: In conclusion, this paper explored the intricate trade-offs between readability, relevance, and factuality in lay summarization. We have highlighted the inherent challenges associated with transforming complex scientific information into an accessible format for non-expert audiences. Our proposed two-stage framework attains state-of-the-art readability metrics; however, this is achieved at the cost of reduced relevance and factuality. These findings emphasize the necessity of striking a balance between readability and content preservation when creating lay summaries.
Future work will focus on integrating simplification and summary generation within a joint optimization framework. This approach aims to overcome the trade-offs identified in our study and enable the generation of high-quality lay summaries without sacrificing readability, relevance, or factuality. More effective lay summarization methods can be developed by considering both simplification and summarization as complementary processes.
Code Availability
Code is available on GitHub: https://github. com/iah3/readability-summarization
Limitations
This study is constrained by limited input token length due to hardware memory limitations and lengthy training times. Even with the LSG attention mechanism’s efficiency, this inadequacy persists in both subtasks. Longer token length could improve summary relevance and factuality. Particularly in the second subtask, where the abstract is absent, this poses a challenge in generating a summary from sections with high information density.

heading: Acknowledgements
text: This research was funded by the NSF CAREER grant 1944247 to C.M, the NIH grant U19AG056169 sub-award to C.M., the Morningside Center for Innovative and Affordable Medicine at Emory University via the Brown Innovation to Market Fund to C.M.

heading: Future Work
text: Future work will focus on integrating simplification and summary generation within a joint optimization framework. This approach aims to overcome the trade-offs identified in our study and enable the generation of high-quality lay summaries without sacrificing readability, relevance, or factuality. More effective lay summarization methods can be developed by considering both simplification and summarization as complementary processes. Code Availability
Code is available on GitHub: https://github. com/iah3/readability-summarization
Limitations
This study is constrained by limited input token length due to hardware memory limitations and lengthy training times. Even with the LSG attention mechanism’s efficiency, this inadequacy persists in both subtasks. Longer token length could improve summary relevance and factuality. Particularly in the second subtask, where the abstract is absent, this poses a challenge in generating a summary from sections with high information density