CSIRO Data61 Team at BioLaySumm Task 1: Lay Summarisation of Biomedical Research Articles Using Generative Models
Year: 2023

Abstract:
Lay summarisation aims at generating a summary for a non-expert audience which allows them to keep updated with the latest research in a specific field. Despite the significant advancements made in the field of text summarisation, lay summarisation remains relatively under-explored. We present a comprehensive set of experiments and analyses to investigate the effectiveness of existing pre-trained language models in generating lay summaries, focusing on the impact of two factors: model size and training data. When evaluating our models in BioLaySumm Shared Task, our submission ranked second for the relevance criteria and third overall among 21 competing teams.1

text: The 22nd Workshop on Biomedical Natural Language Processing and BioNLP Shared Tasks, pages 629–635 July 13, 2023 ©2023 Association for Computational Linguistics

heading: 1 Introduction
text: Text summarisation (Spärck Jones, 1999) is a task in Natural Language Processing (NLP) where its goal is to generate a short and informative summary. Recent developments of summarisation techniques are supported by advancement in pre-trained language models (Devlin et al., 2019; Lewis et al., 2020; Raffel et al., 2020) and availability of largescale datasets which include news articles (Fabbri et al., 2019), scientific publications (Lu et al., 2020) and meeting records (Zhong et al., 2021).
Despite the advances in generic text summarisation, lay summarisation for scientific documents remains less explored. That is, there has been less focus on simplifying technical terms in scientific documents, causing the generated summary to have the same readability level as the input text. Lay summarisation aims to produce a non-technical summary for technical articles (Goldsack et al., 2022). For instance, a lay summary for a scientific article should not contain any technical terms and should be readable and understandable to the public. The lay summary is especially important in
∗This work was partially done when Sim was a summer intern at CSIRO Data61.
1https://github.com/raymondsim/biolaysumm
the biomedical area, where people with no relevant background such as journalists, interdisciplinary researchers, or patients may want to keep updated with recent advances in the field.
We participate in the BioLaySumm shared task (Goldsack et al., 2023) to investigate methods of generating high-quality lay summaries for scientific articles. We present our empirical findings from three different perspectives: (1) model size; (2) data augmentation techniques; and, (3) input length. Our contributions are two-fold: (1) We explore the effectiveness of using pre-trained language models for fine-tuning and data augmentation in lay summarisation, and (2) We investigate the impact of input length on the quality of the generated summary. In the final shared task rankings,2 our approach ranks second for relevance metric and third overall among 21 teams, with a relatively high balance across all metrics.

heading: 2 Related Work
text: Text Summarisation aims to produce a short and concise summary that is representative of information included in input text (Spärck Jones, 1999; Ma et al., 2022a). There are two different types of summarisation: extractive and abstractive. The extractive summarisation model selects important sentences from input text while the abstractive summarisation model has the ability to generate a summary that contains words that do not exist in the input text. There have been promising advances in generating text summary using pre-trained language models (Xiao et al., 2022; Ma et al., 2022b), graph-based summarisation (Yasunaga et al., 2017; Liao et al., 2018; Li et al., 2020; Pasunuru et al., 2021), and hierarchical models (Fabbri et al., 2019; Liu and Lapata, 2019a).
Pre-trained LMs in Summarisation Text summarisation has benefited from pre-trained language
2https://biolaysumm.org/
629
models such as BERT (Devlin et al., 2019), BART (Lewis et al., 2020), and T5 (Raffel et al., 2020). Most studies however used pre-training objectives that are not related to summary generation. Summarisation-specific pre-trained language models are also studied. Liu and Lapata (2019b) introduced BERTSUM, which is capable of generating both extractive and abstractive summaries with a document-level encoder built on top of BERT. Zhang et al. (2020a) proposed PEGASUS, a transformer-based pre-trained language model where summary generation is explicitly included as a pre-training objective. The model is trained to generate abstractive summaries from masked sentences in a document. Here, we leverage the pre-trained LMs not only to generate lay summary but also to generate additional training data.

heading: 3 Approach
text: Backbone Models T5 (Raffel et al., 2020) is a transformer-based text-to-text pre-trained language model. Different tasks, including translation, summarisation, question answering, and classification, are all converted into a text-to-text format. FLANT5 (Chung et al., 2022) is an enhanced version of T5 using instruction fine-tuning. That is, the original T5 model is fine-tuned on tasks phrased as instructions.
We use pre-trained FLAN-T5 models of different sizes (i.e., base, large, and xl),3 with the same instruction across all experiments: “summarise the following article: [DOCUMENT] Summary:”. Since the model should generate long summaries, we use a beam search decoder during inference—a beam width of four—to generate up to 512 tokens. Table 1 shows the training time for fine-tuning these models.
Intermediate-task Pretraining Due to the relatively small size of eLife training set, we also explore the effectiveness of intermediate-task pretraining (Pruksachatkun et al., 2020; Rybinski et al., 2021; Yu et al., 2021) to improve the effectiveness of fine-tuned FLAN-T5 models. Specifically, we first fine-tune FLAN-T5 on a target task-related data set (e.g., PLOS training set and XSum (Narayan et al., 2018) and then on the eLife training set. In addition, we employ an off-theshelf pre-trained language model (i.e., GPT-3.5)
3https://huggingface.co/docs/transformers/ model_doc/flan-t5
to generate additional training data, which will be explained in detail in Section 5.

heading: 4 Datasets and Evaluation Metrics
text: There are two lay summarisation datasets used for the shared task, namely eLife (Goldsack et al., 2022) and PLOS (Goldsack et al., 2022; Luo et al., 2022). eLife4 is a scientific journal for biomedical and life sciences. Publications on eLife contain lay summaries written by expert editors based on the journal article to explain the background and key points of a scientific article to the non-expert. PLOS5 (Public Library of Science) is an openaccess journal, focusing on science and medicine publications where each publication is paired with a lay summary provided by the authors themselves.
The shared task organisers provided training and development sets for model development, and two hidden test sets to score submissions. The descriptive statistics of these datasets can be found in Table 2.
We follow the setup of the shared task to evaluate generated summaries across three aspects: Relevance (ROUGE, BERTScore), Readability (FKGL, DCRS), and Factuality (BARTScore). The aim is to maximise the scores for Relevance and Factuality metrics and minimise scores for Readability metrics. We provide a short description of these metrics and refer readers to (Goldsack et al., 2023) for more details.
ROUGE (Lin, 2004) assesses the quality of generated summaries by comparing generated summaries to gold summaries and counting the number of overlapping n-grams, word sequences, and word pairs between summaries.
BERTScore (Zhang et al., 2020b) uses contextual embedding in BERT (Devlin et al., 2019) to compute the similarity score for each token between the source document and generated summary and average them to obtain the overall effectiveness metric of a summarisation model.
FKGL (Flesch-Kincaid Grade Level) (Flesch, 1948) indicates difficulty in reading a passage in English based on two factors: the average sentence length and the average number of syllables per word.
DCRS (Dale-Chall Readability Score) (Chall and Dale, 1995) is a score based on occurrence of words unknown to most 4th-grade students (in the US education system). A lower score indicates higher readability.
BARTScore (Yuan et al., 2021) defines the evaluation of generated text as a text generation problem that uses BART as its backbone. It utilises the generation probabilities of BART to measure the quality of a sentence.

heading: 5 Experimental Results

heading: Impact of Scaling the Model Size
text: We empirically compare the effectiveness of three publicly available pre-trained FLAN-T5 models which differ only in the number of parameters: FLAN-T5-base (250M), FLAN-T5-large (780M), and FLAN-T5-xl (3B). We fine-tune these FLANT5 models on the eLife—for 25 epochs—and
PLOS—for 5 epochs—training sets respectively.6 The fine-tuned models are then evaluated on the corresponding development sets.
Table 3 shows that larger models always result in better relevance scores (i.e., ROUGE scores and BERTScore), but not necessarily on readability scores (i.e., FKGL and DCRS) or factuality score (i.e., BARTScore). In fact, the smallest model— FLAN-T5-base—achieves the best BARTScore and DCRS scores, when evaluating eLife and PLOS development sets.
We use FLAN-T5-xl in the following experiments as default, unless otherwise specified.

heading: GPT-3.5 to Generate Additional Training Data
text: One observation from Table 3 is that evaluation scores on the eLife sets fall behind those on the PLOS sets. We hypothesise that this may be attributed to either the different characteristics of these two datasets (Table 2) or the relatively small size of the eLife training set (4346 examples vs. 24773 examples in PLOS training set). In order to understand the trade-off between domain and training data size, we conduct a cross-dataset evaluation. That is, we evaluate the effectiveness of the model that is trained using the PLOS training set, on the eLife development set. In addition, we use the combination of the eLife and PLOS training sets to train the model, which is then evaluated on the eLife development set. Results in the upper part of Table 4 show that although the PLOS set and the combined set contain a much larger number of training examples than the eLife training set, the models trained on the former two under-perform compared to the model trained only on eLife by a large margin (all three models are evaluated on the eLife development set).
6Because the eLife training set is smaller than PLOS, we choose a larger number of training epochs for it.
We also investigate whether intermediate-task pretraining can be used to improve the model training when only a relatively small size of training data (i.e., eLife) is available. More specifically, we first fine-tune FLAN-T5-xl on a source summarisation dataset (source), and then continue
fine-tuning the model on the eLife training set (target). We consider four possible sources: (1) PLOS, which focuses on the task of lay summarisation with data sampled from different journals; (2) XSum (Narayan et al., 2018), which focuses on news articles summarisation; we choose it because of its relatively large training data size and simpler language used in its summaries; (3) GPT3.5 (P), where for each target training example, a paraphrased summary is generated using OpenAI API (gpt-3.5-turbo, https://platform.openai. com/docs/models/gpt-3-5);7 and, (4) GPT-3.5 (S), where GPT-3.5 is asked to generate summary for each document in the target training set.8 Results in Table 4 show that all sources can help the model with improved relevance scores (ROUGE and BERTScore), demonstrating the benefits of using additional training data. On one hand, GPT-3.5 (P) outperforms other sources in all four relevance metrics, although its size is much smaller than PLOS and XSum training set. On the other hand, this benefit becomes unclear regarding the readability and the factuality metrics. For example, the model effectiveness in terms of readability scores (FKGL and DCRS) decreases for all the sources, and the model trained on the combination of eLife and PLOS achieves the highest BARTScore (factuality), whereas other models achieve very close results. Considering the low cost of using GPT-3.5 to generate synthetic data, it is a promising direction to use it for data augmentation and even for data annotation (Wang et al., 2022).
7Prompt template: “Paraphrase the following paragraph: [Summary] Paraphrased:”
8Prompt template: “Summarise the following document using plain text: [Input document] Summary:”

heading: Impact of Input Document Length on Generated Summary
text: We use the input sequence length of 512 as a baseline and measure the relative improvement or decline of different scores as we change to different input lengths of test examples. Figure 1a shows that the model fails to generate a relevant summary (decreased ROUGE scores) if the input document is truncated to a shorter length. In contrast, taking a longer document as input to the summarisation model slightly improves the relevance scores. However, they decrease again when the input length is more than 2000 tokens, which potentially highlights the difficulty of capturing long-range contextual dependencies. Results on readability and factuality metrics (Figure 1b) show that these metrics are less affected by the length of the input document. In fact, the model can generate fluent text—with good BARTScore DCRS and scores— even with just a few tokens provided as input.

heading: 6 Conclusions
text: We present our approach to the lay summarisation for scientific documents in BioNLP 2023 shared task. Our approach utilises GPT-3.5 to generate additional training data and pre-trained FLAN-T5 in generating a lay summary for scientific documents. Our results show that extra data generated from a generative model can boost the effectiveness of a summarisation model to a certain degree, especially in terms of relevance metrics. In addition, we showed that input length impacts the relevance of generated summary, but has no obvious impact on the readability and factuality metrics. Future work could focus on developing a summarisation model that optimises readability and factuality of generated summary.

heading: Limitations
text: In this work, we used GPT-3.5 to generate additional training examples and showed that it helps improve the relevance (ROUGE and BERTScore) of the generated summary. However, we did not explicitly analyse the quality of generated examples to check whether or not they are faithful and factually correct which could lead to the same problem in generated summaries. To obtain better training examples, we could use faithful or factuality metrics to assess generated training examples and then use the post-editing method or human evaluation
to remove unfaithful content, which we leave for future research.

heading: Acknowledgements
text: This work is supported by The Commonwealth Scientific and Industrial Research Organisation (CSIRO) Precision Health Future Science Platform (FSP). Experiments were undertaken with the assistance of resources and services from the National Computational Infrastructure (NCI), which is supported by the Australian Government.

heading: Future Work
text: This work is supported by The Commonwealth Scientific and Industrial Research Organisation (CSIRO) Precision Health Future Science Platform (FSP). Experiments were undertaken with the assistance of resources and services from the National Computational Infrastructure (NCI), which is supported by the Australian Government