IKM_Lab at BioLaySumm Task 1: Longformer-based Prompt Tuning for Biomedical Lay Summary Generation
Year: 2023

Abstract:
This paper describes the entry by the Intelligent Knowledge Management (IKM) Laboratory in the BioLaySumm 2023 task1. We aim to transform lengthy biomedical articles into concise, reader-friendly summaries that can be easily comprehended by the general public. We utilized a long-text abstractive summarization longformer model and experimented with several prompt methods for this task. Our entry placed 10th overall, but we were particularly proud to achieve a 3rd place score in the readability evaluation metric. Our code is available at https://github.com/IKMLab/ BioLaySumm.

text: The 22nd Workshop on Biomedical Natural Language Processing and BioNLP Shared Tasks, pages 602–610 July 13, 2023 ©2023 Association for Computational Linguistics

heading: 1 Introduction
text: The BioLaySumm 2023 Task 1: Lay Summarization of Biomedical Articles (Goldsack et al., 2023) is to generate lay summaries for two biomedical datasets (Goldsack et al., 2022), PLOS and eLife, respectively. The aim of this task is to generate simplified and easy-to-understand summaries of the original articles that can be readily comprehended by the general public. Figure 1 provides an information of this task dataset, showcasing examples that highlight the importance of not only generating abstracts for the text but also simplifying the content. In the examples, the bold vocabulary in the article text represents biomedical terminology that may be difficult for the general public to understand. Therefore, it is crucial to transform such terminology into language that is more accessible and easily understood by a lay audience, as depicted by the bold vocabulary in the lay summary. This transformation plays a vital role in ensuring that the generated lay summaries are both comprehensible and effectively convey the key information to non-expert readers.
The average length of articles in the eLife dataset is 8,442 words, while for the PLOS dataset, it is 5,864 words. (Table 1.). Using a state-of-the-art
summary generation model with good performance on shorter texts would not be sufficient to capture all the important information in these lengthy articles, as the model would not be able to process such a large number of tokens. In order to solve the problem of dealing with long texts, we utilized models like Longformer (Beltagy et al., 2020) as the main framework in our competition.
In our implementation, we go beyond using just the article field and also incorporate the information from the keyword and heading fields. By extracting and utilizing these additional fields during the training process, our goal is to generate lay summaries that are more informative, accurate, and easily comprehensible to a general audience. This approach allows us to leverage the rich information contained in the keyword and heading fields,
602
thereby enhancing the quality and effectiveness of the generated summaries.

heading: 2 System Overview
text: Our system employs a pre-trained longformer (Beltagy et al., 2020) as the primary architecture and implements diverse prompt formats for the keyword and heading fields in the dataset during training. In the following sections, each component of the system will be described in detail.

heading: 2.1 Longformer
text: In our implementation, we used the pre-trained LED model (Beltagy et al., 2020) available on HuggingFace1 (Wolf et al., 2020) to fine-tune each of the two datasets because LED accepts a maximum sequence length of 8,192 tokens, which shows potential for fitting the two datasets. As we observe the data statistics in Table 1, there are variations in the average length of the lay summaries between the eLife and PLOS datasets. Considering that the average length of eLife’s lay summaries is 348 words, we set a maximum output sequence length of 512 during the decoding process. This adjustment also accommodates some lay summaries in the PLOS dataset that exceed the limit of 256 words.
We trained the model for 2 epochs on each dataset. Compared with our own fine-tuning experiments with the pre-trained BART model (Lewis et al., 2020), we found that the LED model performed better during the fine-tuning stage, which was in line with our expectations.

heading: 2.2 Prompt
text: We drew inspiration from existing research on optimized summarization techniques. Controlled tokens (Luo et al., 2022) to regulate the form and
1https://huggingface.co/models
style of summary generation, using explicit tokens (Martin et al., 2020) to improve the effect of simplification, and enhancing the learning effect of the model by adding prompts (Zhang et al., 2022).
Based on the aforementioned method, we examined the various information provided in both datasets and discovered the existence of a “keywords” column, which describes the topic of the article, as well as a “headings” column. We utilized the information contained in these two fields to guide our prompt form design.
Keywords: To leverage the information provided in the keywords column, we devised two types of prompts. The first prompt form involved concatenating the keywords column with the article “Keywords (k1, k2, ..., km) summarize: ", The second prompt form is to replace the [KEYWORDS] and [LAY_SUM] placeholders in the prompt with special tokens “[KEYWORDS] (k1, k2, ..., km) [LAY_SUM]: ", this was done to highlight the importance of these specific tokens and make it easier for the model to differentiate them during training.
Headings: The title of each section in the text is represented by a heading, such as “Abstract” ,“Methods” etc., and each section is separated by “\n" in the article. To help the model better understand the correlation between each section, we added heading prompts. Compared to the keyword prompts, the form we designed using headings is h1 Sections1, h2 Section2,... and hi ∈ H(n = 1, 2, ..., n) where H is a list of headings names.

heading: 2.3 Prompt Tuning
text: During the prompt tuning stage, we implemented the prompt forms described earlier by converting them into string format. These prompts were then concatenated with the article text, creating a connected text that incorporated the relevant prompts. This connected text was subsequently used as input for the Longformer model during the prompt tuning process. By integrating the prompts directly into the text, we aimed to guide the model towards better understanding and generating higher quality lay summaries that aligned with our desired objectives.

heading: 3 Experiments

heading: 3.1 Evaluation Metrics
text: In this competition, three evaluation aspects have been defined to assess the effectiveness of sum-
marization generation and simplification, and each aspect comprises multiple automatic metrics:
• Relevance: ROUGE-1 (R-1), ROUGE-2 (R-2), ROUGE-L (R-L) (Lin, 2004), and BERTScore (Zhang et al., 2019).
• Readability: Flesch-Kincaid Grade Level (FKGL) (Kincaid et al., 1975) and Dale-Chall Readability Score (DCRS) (Chall and Dale, 1995).
• Factuality: BARTScore (Yuan et al., 2021). The evaluation model is provided by the official.
Of the three evaluation aspects, excluding readability, a lower index value is indicative of better performance, while higher scores in relevance and factuality are desirable.

heading: 3.2 Model Analysis
text: Considering the substantial size of the article, prioritizing the implementation of summarization over simplification is of utmost importance. Hence, the initial implementation involves fine-tuning both datasets within a range of prominent abstract summarization models, including the LED model of interest.
The results of various metrics are presented in Table 2, where the average of the two validation datasets evaluated separately is reported. The results indicate that for pure fine-tuning, the LED model performs as expected. Although it exhibits slightly lower readability compared to the T5 model (Raffel et al., 2020), other evaluation metrics show significant improvements. Hence, a minor decrease in readability metrics is deemed acceptable. Therefore, we chose the LED model as our primary architecture for further experimentation.

heading: 3.3 Prompt Analysis
text: This section focuses on the utilization of various prompts to fine-tune the pretrained LED model for handling long texts, as discussed in Section 2.2.
Our main objective was to improve the quality of summaries and simplify the text through prompt tuning. Additionally, we conducted a comparative analysis with the BART model, employing similar prompts simultaneously, to comprehensively evaluate the effectiveness of the LED model after prompt integration.
Upon analyzing the prompt tuning results of the LED model, we observed that when using the PLOS dataset, the overall scores of the relevance and factuality metrics remained relatively stable across different prompt methods compared to the absence of prompt tuning. Moreover, we found that both readability metrics (FKGL and DCRS) exhibited simultaneous improvements, resulting in the simplification of the text, the results show in Table 3 (with “ST” representing “Special Tokens”).
The final evaluation results of the two validation datasets, as presented in Table 4, demonstrate that the performance of the prompt utilizing special tokens to represent KEYWORDS and LAY_SUM is comparable to that of the prompt without special tokens. The scores only exhibited slight fluctuations, and the performance of the seven evaluation metrics was superior to that of using headings as a prompt. In terms of model selection based on the prompts, the LED model consistently outperformed the BART model in overall performance, aligning with our initial expectations.

heading: 3.4 Keyword Analysis
text: In this section, we analyze the keywords field in the two datasets to discuss the potential negative impact of keywords on our method.
1. Coarse keywords in the datasets. Through observations on the datasets, we found that the granularity of the keyword field is relatively coarse. The words used in this field typically represent the overall theme or topic of the entire article, such as “biochemistry” and “cell biology”. These keywords often fail to capture the intricate details of an article and may lead to lower scores on ROUGE-1 and ROUGE2 in Table 4, compared with the pre-trained LED in Table 2.
2. Missing keywords in PLOS. The number of empty keywords field over the data points in the PLOS dataset is much higher than that in the eLife dataset (Table 5), which may cause inconsistent keyword prompts for the instances of the two datasets.
3. Low keyword diversity in eLife. The number of unique keywords and the average number of keywords per example in the PLOS dataset are much higher than in the eLife dataset, as listed in Table 5. In addition, eLife provides keywords with an even coarser granularity for each data point than PLOS. These may further hinder our model from capturing the article details in eLife.

heading: 3.5 Headings Analysis
text: Analysis of headings indicates that prompt tuning has a relatively weaker impact compared to keywords. The evaluation metrics show a slight increase only in the FKGL readability metric and BARTscore, while other indicators decline to some extent. Our observation reveals that headings are more complex than anticipated. Although common headings like Abstract, Introduction, Method,
and Discussion are present, there are also unique and specific headings tailored to individual articles, such as “Bacterial Films” or “The Roles of VSG N-linked Oligosaccharides”. These headings provide less generalizable information compared to keywords. Additionally, variations in paragraph information across articles may hinder the model’s understanding of paragraph importance. Consequently, headings offer less comprehensive information compared to keywords.

heading: 4 Submission Options

heading: 4.1 Submission 1
text: To maintain consistency, we used the pure finetuned LED model as the initial submission (the LED model in Table 2) and compared its results with the evaluation results of the local validation dataset. We also conducted an analysis to determine whether the validation data and testing data had similar properties.

heading: 4.2 Submission 2
text: For the second submission, we selected the model with the best prompt tuning results. Although the performance of special tokens was better in terms of relevance indicators, the improvement rate was insignificant, and the readability metric showed a significant drop. Therefore, we decided to use the keywords prompt without special tokens as our second submission (that is, LED keywords w/o ST model in Table 4).

heading: 4.3 Results
text: We present the evaluation scores for each metric in Table 6 for both of the submissions and show the generation comparison in Appendix A. Based on the overall judgment, Submission 1 outperformed Submission 2 and was chosen as our final submission for comparison with other teams. Our system achieved an overall 10th ranking and placed third in the readability metrics.

heading: 5 Conclusion
text: Overall, fine-tuning the pre-trained LED model for lay summary generation tasks on the PLOS and eLife datasets produced satisfactory results. Surprisingly, the model demonstrated excellent scores on readability metrics, indicating its effectiveness in text simplification. However, the use of information design prompts such as keywords and headings did not improve the LED model’s ability to learn and generate better lay summaries as expected, and most of the evaluation metrics declined.
The final results of our experiments have revealed the need for further investigation and research into the prompt strategy that was initially intended to enhance the generation of lay summaries. We aim to determine the underlying causes of the decline observed in the evaluation metrics. Possible factors contributing to this decline include inadequate quality or insufficient differentiation of keywords from the content or abstract, excessive links or correlations within the keywords, or inconsistencies in the formatting of headings across articles, which may hinder the model’s ability to grasp the importance of each paragraph’s content. To address these issues and improve the effectiveness of the prompts, we will explore and develop new methods in future research.
Limitations
The performance of the model heavily relies on the quality and relevance of the keywords and headings provided in the dataset. If the dataset lacks rigor during the column labeling stage or if the information in these fields is inadequate, it can lead to a
decrease in the overall effectiveness of the model. Moreover, incorporating additional prompt tokens may introduce length constraints, potentially resulting in article truncation and the loss of crucial information.

heading: A Appendix
text: We provide an example of the eLife validation set for Submission 1 in Table 7 and Submission 2 in Table 8, with the following highlighting guidelines:
1. Factually consistent to Gold and fluent.
2. Factually consistent to Gold but not fluent.
3. Facutally inconsistent to Gold but fluent.
4. Facutally inconsistent to Gold and not fluent
The sentences in the gold article corresponding to the relevant generated sentences are highlighted in orange. All the highlights are initially provided by ChatGPT (GPT-4) and then manually reviewed by the authors. Note that the second category (purple) is not found in this example.

heading: Future Work
text: To address these issues and improve the effectiveness of the prompts, we will explore and develop new methods in future research. Limitations
The performance of the model heavily relies on the quality and relevance of the keywords and headings provided in the dataset. If the dataset lacks rigor during the column labeling stage or if the information in these fields is inadequate, it can lead to a
decrease in the overall effectiveness of the model. Moreover, incorporating additional prompt tokens may introduce length constraints, potentially resulting in article truncation and the loss of crucial information