MDC at BioLaySumm Task 1: Evaluating GPT Models for Biomedical Lay Summarization
Year: 2023

Abstract:
This paper presents our approach to the BioLaySumm Task 1 shared task, held at the BioNLP 2023 Workshop. The effective communication of scientific knowledge to the general public is often limited by the technical language used in research, making it difficult for non-experts to comprehend. To address this issue, lay summaries can be used to explain research findings to non-experts in an accessible form. We conduct an evaluation of autoregressive language models, both general and specialized for the biomedical domain, to generate lay summaries from biomedical research article abstracts. Our findings demonstrate that a GPT-3.5 model combined with a straightforward few-shot prompt produces lay summaries that achieve significantly higher relevance and factuality compared to those generated by a fine-tuned BioGPT model. However, the summaries generated by the BioGPT model exhibit better readability. Notably, our submission for the shared task achieved 1st place in the competition.

text: The 22nd Workshop on Biomedical Natural Language Processing and BioNLP Shared Tasks, pages 611–619 July 13, 2023 ©2023 Association for Computational Linguistics

heading: 1 Introduction
text: Effective communication of scientific ideas is essential for sharing research findings with the general public. While scientific publications serve as an important tool for scientists to share their work, they are primarily intended for other researchers. The use of technical language and the required background knowledge makes these articles difficult for the general public to comprehend, limiting accessibility and research impact (Kuehne and Olden, 2015). The goal of lay summarization is to generate a concise summary of technical texts without using technical language so it can be understood by non-experts. Although certain publishers require authors to provide a lay summary alongside their manuscripts, not all publishers have this requirement, and historical publications are unlikely to include such summaries. Therefore, the devel-
opment of effective automatic lay summarization becomes crucial in bridging this communication gap (Guo et al., 2021).
GPT-style autoregressive language models have demonstrated impressive ability for natural language generation (NLG) tasks including summarization (Radford et al., 2018; Radford et al., 2019). These models are pre-trained on large corpora and can leverage knowledge gained from this data to generate text. Recent research has rapidly advanced the performance of GPT-style models by substantially scaling the size of these models and their training corpora (Brown et al., 2020), along with instruction tuning and reinforcement learning with human feedback (Ouyang et al., 2022).
Researchers have evaluated GPT models on biomedical-specific natural language understanding (NLU) tasks and found that they underperform due to a lack of domain knowledge (Jimenez Gutierrez et al., 2022; Moradi et al., 2022). Recent research by Luo et al. (2022a) aimed to alleviate this shortcoming by proposing BioGPT, a model based upon GPT-2 (Radford et al., 2019) but specifically trained on biomedical text from PubMed1. The authors demonstrate that BioGPT can significantly outperform GPT-2 on biomedical NLU tasks; however, there is currently no evaluation on biomedical natural language generation (NLG) tasks. Therefore, the objective of this work is to evaluate the ability of general-purpose and domainspecific GPT models to generate lay summaries of biomedical research articles as part of the BioLaySumm challenge (Goldsack et al., 2023).
This approach raises an issue. The attention mechanism used in many GPT-style models has quadratic memory scaling with the length of a sequence, which limits the maximum length of input text that can be processed. At the time of writing, the average length of a research article exceeded this limit. Journal articles already contain author-
1https://pubmed.ncbi.nlm.nih.gov/
611
written abstracts that provide a technical summary of the paper and previous work by Goldsack et al. (2022) demonstrated the importance of the abstract in lay summarization. Therefore, we focused on using only the article abstract as the input and leveraging the domain knowledge of these models to generate lay summaries.
We present results for this method for zero-shot and fine-tuned BioGPT and GPT-2 models. Additional experiments are conducted using GPT-3.5 variants accessed via the OpenAI API and we compare their performance with the smaller - but domain-specific - BioGPT model. Our findings indicate that a GPT-3.5 model, combined with a straightforward few-shot prompt, achieves the best performance among the evaluated models. This approach secured 1st place in the shared task.

heading: 2 Background

heading: 2.1 Autoregressive language models
text: Autoregressive language models are trained using a conditional generation task, where the model is trained to predict the next token in a sequence given all previous tokens in the sequence. The model is given a sequence of tokens x of length T from a training corpus of N sequences, where each sample consists of a sequence of numeric tokens that map to a natural language vocabulary. The model’s training objective is then formulated as maximizing the log-likelihood across the training corpus, given the model’s parameters θ:
L(θ) = N∑
i=1
Ti∑
t=1
logPθ(xi,t|xi,1:t−1) (1)
The parameters θ are learned through backpropagation and stochastic gradient descent. Once this first phase of training is completed, often referred to as pre-training, the model can be fine-tuned for a specific task using additional data. To achieve this, the input and expected output for the model needs to be converted into a format that is compatible with the training objective in Equation 1. This requires framing the task in natural language and formatting the training data to match this structure.
A popular approach (Brown et al., 2020), and the method used in Luo et al. (2022a), is to train the model on sequences consisting of the information required for the task, called the source, along with a natural language description of the task, called the prompt, and the expected output, called the
target. For summarization tasks, this task can then be structured as the text to be summarized as the source, a description of the summarization required as the prompt and an example summary as the target. At inference time, only the source and prompt are provided as input and the model generates the missing summary.

heading: 2.2 Zero-shot and few-shot prompting
text: The simplest method for querying a language model - known as zero-shot prompting - involves passing a single prompt alongside the source and requires the model to immediately generate the target. Few-shot prompting is a form of in-context learning (Dong et al., 2023) that builds upon zero-shot prompting by providing several source texts and target examples along with a prompt instruction. For lay summarization, these examples would be abstract and lay summary pairs. See Appendix A for example zero and few-shot prompts. While zero and few-shot prompting can produce impressive results (Brown et al., 2020), fine-tuning a model to perform a task is usually a more powerful approach but requires significantly more resources and access to the model parameters.

heading: 3 Experiments

heading: 3.1 Dataset
text: The dataset for the shared task is outlined in Goldsack et al. (2022) and Luo et al. (2022b). It consists of two sets of biomedical articles from the eLife and Public Library of Science (PLOS) journals. Table 1 lists the dataset statistics.
For each article, the entire text was provided along with an expert-created lay summary of the article. The articles were split based on the sections contained within and the abstract was extracted.

heading: 3.2 Metrics
text: The generated summaries were evaluated according to three criteria: relevance, readability, and factuality. Relevance was measured by the ROUGE-1, ROUGE-2, ROUGE-L (Lin, 2004; Lin and Hovy, 2003; Lin and Och, 2004) and BERTScore (Zhang et al., 2020) metrics. Readability was measured using the Flesch-Kincaid Grade Level (FKGL) (Kincaid et al., 1975) and Dale-Chall Readability Score (DCRS) (Chall and Dale, 1995) metrics. The BARTScore metric (Yuan et al., 2021) - computed using a model fine-tuned for the shared task by the organisers - was employed to measure factuality.

heading: 3.3 Experimenting with GPT models
text: We fine-tuned and evaluated the open-source GPT models using the PyTorch v2.0.1 (Paszke et al., 2019) and Huggingface Transformers v4.28.1 (Wolf et al., 2020) libraries. The following pre-trained models were experimented with: GPT-2MEDIUM2, GPT-2XL3, BioGPT4 and BioGPTLARGE5
To create the training samples used to fine-tune the models, the abstract was used as the source input, a prompt of "Explanation: " was used and the provided lay summary of the article was used as the target. The eLife and PLOS training datasets were combined to form a single training dataset. During inference, the model was given an abstract as the source along with the same prompt used in training. The text generated by the model after the prompt was extracted and used to evaluate the model. Contrary to the fine-tuning work carried out in Luo et al. (2022a), we focused solely on using manually designed prompts instead of the prefix-tuned soft prompts (Li and Liang, 2021) the authors used.
It was decided that due to the resource and energy requirements in training large language models, no hyperparameter optimisation was to be performed for any of the models being evaluated and the same hyperparameters were to be used for all models. Fine-tuning training for all models was performed across 4 Nvidia V100-SXM2 32GB GPUs with PyTorch DDP (Li et al., 2020), using gradient checkpointing and FP16 precision. Each model was fine-tuned for a maximum of 4 epochs with
2https://huggingface.co/gpt2-medium 3https://huggingface.co/gpt2-xl 4https://huggingface.co/microsoft/biogpt 5https://huggingface.co/microsoft/BioGPT-Large
a total batch size of 32. The Adafactor optimiser (Shazeer and Stern, 2018) was used with a peak learning rate of 2e-5. During training, a weight decay of 0.01 was applied and the gradient normal was clipped to a maximum value of 1. The learning rate followed a cosine-decay schedule, with 10% of the total training steps used as warm-up steps. Validation metrics were calculated after every 50 steps, with training stopped early if the validation loss did not decrease after 3 consecutive evaluation steps.
For text generation, contrastive search sampling (Su et al., 2022; Su and Collier, 2023) was used, with hyperparmeters k = 6 and α = 0.5 for a maximum of 768 new tokens. Again, no hyperparameter selection was carried out for these values and they were set based on manual evaluation of selected generations from the validation set. During inference, all text generation was carried out on a single V100-SXM2 32GB GPU using FP16 precision.

heading: 3.4 Experimenting with the OpenAI API
text: GPT-3.5 exhibits strong few-shot performance across a range of natural language processing tasks (Ouyang et al., 2022) and the OpenAI API enables users to easily query their models for a fee. Given the model’s strong performance in NLG tasks and the ease of using the API, we experimented with two GPT-3.5 models6 for lay summarization: gpt3.5-turbo (the model used by ChatGPT7, hereafter referred to as the chat model) and text-davinci-003 (the most powerful instruction-tuned model, hereafter referred to as the instruct model). OpenAI’s documentation suggests the chat model offers comparable performance to the instruct model at a fraction of the cost. For this reason we performed the majority of our experiments using the chat model.
We experimented with two different prompting schemes: a simple zero-shot prompt, and few-shot abstract-lay summary pair prompts. When generating abstract-lay summary pair prompts we included as many as would fit in the API call’s token limit when combined with the abstract to be summarized and the generated summary (4096 and 4097 tokens for the chat and instruct models, respectively). The length of the generated summary was limited to the maximum summary length observed in the relevant training data set (Table 1). This resulted in eLife and PLOS few-shot prompts containing ∼3-5 and
6https://platform.openai.com/docs/models/gpt-3-5 7https://openai.com/blog/chatgpt
∼3-7 example pairs respectively. The temperature parameter - which controls the entropy of the token probability distribution during text generation - was tuned using the eLife dataset only to reduce costs. The performance was consistent for each temperature setting; we selected a temperature of 0.3 for the final evaluation.

heading: 4 Results and Discussion
text: Table 2 lists the combined eLife and PLOS validation set performance metrics, with the few-shot instruct model performing best overall. As expected, fine-tuning the BioGPT and GPT-2 models greatly enhances the relevance, readability, and factuality of the generated summaries. Both finetuned BioGPT models outperform the GPT-2 models across nearly all metrics, demonstrating the beneficial impact of leveraging biomedical domain knowledge gained during pre-training for text generation. The smaller BioGPT and GPT-2MEDIUM models exhibit better readability metrics than the large model variants, however the relevance and factuality metrics are significantly worse. Interestingly, the zero-shot performance of both BioGPT models is considerably inferior to that of the zeroshot GPT-2 models. This discrepancy may be attributed to the smaller dataset used for pre-training the BioGPT models which could reduce the zeroshot generalisation performance of the model, but this would need further investigation.
Few shot prompting of the instruct model significantly improved its performance and this model achieved the best results for the factuality and relevance metrics. The remarkable increase in performance of the few-shot versus the equivalent zeroshot models demonstrates the large language models’ ability for in-context learning. Interestingly, the performance of the fine-tuned BioGPTLARGE model is not far off the text-davinci-003 despite being 1/100th the size of the GPT-3.5 model and even outperformed chat and instruct models in their zero-shot setting. This shows the benefit of domain knowledge for domain-specific natural language generation tasks but also alludes to potential performance gains of scaling up the size of domainspecific language models.
The chat model performed significantly better than the instruct model when using the zero-shot prompt. This may be because the zero-shot prompt is in a conversational style, which the chat model was trained for, or additional training to the chat
model that the instruct model did not receive. We are limited to conjecture as the training details and differences of the chat model from the instruct model are largely unknown as they have not been published by OpenAI.
Due to the superior performance on the validation set scores, we chose the instruct model for the final test evaluation. The test set scores are listed in Table 3 alongside the fine-tuned BioGPTLARGE test scores and the test scores of the BART model proposed by the shared task organisers (Goldsack et al., 2022). The few-shot instruct model outperforms both of these models, achieving either the best or second best score for nearly all metrics. However, the fine-tuned BioGPTLARGE model did generate summaries with the best readability at the cost of worse factuality. We submitted the few-shot instruct model test set results to the shared task competition and out of 21 participants, the model came in 1st place overall. Across the evaluation criteria, the model came in 3rd place for relevance and factuality but only achieved 10th place on readability.
It would be valuable to compare the performance of the proposed approaches with other recently open-sourced large language models such as the biomedical-domain BioMedLM8, the scientificdomain Galactica (Taylor et al., 2022) or the general-domain LLaMA (Touvron et al., 2023). Another interesting future research avenue is available with the longer context models. The emergence of models such as GPT-4 (OpenAI, 2023), PALM-2 (Anil et al., 2023) and Claude9 offers significantly increased context lengths compared to the models assessed in this paper. Using these models, it would be possible to input more sections such as the introduction from a paper into the few-shot prompts, potentially leading to substantial enhancements in performance as these sections provide additional background information. Furthermore, the performance of the BioGPTLARGE model has room for improvement. All hyperparameters were fixed during training of the model due to resource constraints so tuning of the training and generation hyperparameters may benefit model performance. The hyperparameters for all models were also not tuned for a specific dataset and separately tuning these for each dataset may improve summary quality. Advanced prompting techniques have been
8https://huggingface.co/stanford-crfm/BioMedLM 9https://www.anthropic.com/index/introducing-claude
shown to improve performance over manually designed prompts and may enhance performance (Liu et al., 2022; Li and Liang, 2021).
While the capability shown in these results is intriguing, this method is not yet mature enough to be relied upon to generate lay summaries. GPT models are known to hallucinate factual information (Ji et al., 2023) and detecting these hallucinations in text is difficult for both machines and the untrained eye. If the general public were to depend on these generated lay summaries for comprehending scientific research, the confident dissemination of erroneous information would be actively detrimental. Further work is required to verify the factuality of the generated lay summaries using human experts before this approach could be safely used.

heading: 5 Conclusion
text: This paper presents an evaluation of generalpurpose and biomedical domain-specific autoregressive language models for generating lay summaries from the abstracts of scientific articles for the BioLaySumm shared task at BioNLP 2023. Our findings demonstrate that the biomedical domainspecific model, BioGPT, outperforms generalpurpose GPT-2 models when fine-tuned for lay summary generation. We also explored the effectiveness of zero and few-shot prompting for generating lay summaries using OpenAI’s GPT-3.5
models. While the zero-shot performance of these models is worse than the fine-tuned BioGPTLARGE model, we discovered that using the text-davinci003 coupled with a few-shot prompt yielded the best results among all the language models tested. This approach was selected as our submission for the shared task, achieving the overall 1st place submission and 3rd place in both relevance and factuality metrics. To further enhance the performance of this approach, we anticipate that advanced prompting methods, evaluation of additional models, and utilization of models with longer context lengths could be beneficial. Despite the promising results obtained in this study, it is essential to conduct further research to validate the factuality of the generated lay summaries using human experts before practical application.
Limitations
Our best results were obtained using a few-shot prompt of the text-davinci-003 model from OpenAI. While the technical barriers to this method are very low due to the ease of use of the model API, the cost of querying the API can become prohibitively expensive and this limited our own experiments with the model10. This cost would rise significantly more if the text-davinci-003 was finetuned to perform these lay summaries. If this is a concern, then using the open-sourced BioGPT models may be beneficial. It should be noted that performing the fine-tuning process is itself expensive and requires access to high-end GPUs. Practitioners should investigate parameter-efficient finetuning techniques (Hu et al., 2021) if access to these GPUs is an issue.

heading: A Example prompts

heading: Future Work
text: Another interesting future research avenue is available with the longer context models. The emergence of models such as GPT-4 (OpenAI, 2023), PALM-2 (Anil et al. , 2023) and Claude9 offers significantly increased context lengths compared to the models assessed in this paper. Using these models, it would be possible to input more sections such as the introduction from a paper into the few-shot prompts, potentially leading to substantial enhancements in performance as these sections provide additional background information. Furthermore, the performance of the BioGPTLARGE model has room for improvement. All hyperparameters were fixed during training of the model due to resource constraints so tuning of the training and generation hyperparameters may benefit model performance. The hyperparameters for all models were also not tuned for a specific dataset and separately tuning these for each dataset may improve summary quality. Advanced prompting techniques have been
8https://huggingface. co/stanford-crfm/BioMedLM 9https://www. anthropic. com/index/introducing-claude
shown to improve performance over manually designed prompts and may enhance performance (Liu et al. , 2022; Li and Liang, 2021). While the capability shown in these results is intriguing, this method is not yet mature enough to be relied upon to generate lay summaries. GPT models are known to hallucinate factual information (Ji et al. , 2023) and detecting these hallucinations in text is difficult for both machines and the untrained eye. If the general public were to depend on these generated lay summaries for comprehending scientific research, the confident dissemination of erroneous information would be actively detrimental. Further work is required to verify the factuality of the generated lay summaries using human experts before this approach could be safely used