Saama Technologies at BioLaySumm: Abstract based fine-tuned models with LoRA (2024)

Abstract: Lay summarization of biomedical research articles is a challenging problem due to their use of technical terms and background knowledge requirements, despite the potential benefits of these research articles to the public. We worked on this problem as participating in BioLaySumm 2024. We experimented with various fine-tuning approaches to generate better lay summaries for biomedical research articles. After several experiments, we built a LoRA model with unsupervised fine-tuning based on the abstracts of the given articles, followed by a postprocessing unit to take off repeated sentences. Our model was ranked 3rd overall in the BioLaySumm 2024 leaderboard. We analyzed the different approaches we experimented with and suggested several ideas to improve our model further.

text: Proceedings of the 23rd Workshop on Biomedical Language Processing, pages 786–792 August 16, 2024. ©2024 Association for Computational Linguistics
786

heading: 1 Introduction
text: While many academic publications in the biomedical field can potentially benefit a wide readership including many non-experts, their accessibility is often limited by their use of technical terms and relatively sophisticated expressions. Therefore the summarization of biomedical research articles is an interesting and important task that can benefit the general public, and BioLaySumm 2024 (Goldsack et al., 2024) aims to solve this question by adopting techniques of NLP. BioLaySumm asks participants to suggest models that summarize the biomedical articles based on the PLOS and eLife datasets (Goldsack et al., 2022) composed of original research articles and lay summaries written by experts.
In this paper, we explain our approaches to the BioLaySumm 2024 in detail. To generate better lay summaries, we experimented with multiple finetuning approaches with LoRA based on the abstract part of the biomedical research papers. As a result of a series of experimentations, we concluded
that our best-performing model is the unsupervised fine-tuned model with LoRA followed by a postprocessing unit that chops off repeated sentences in the raw predictions. At the end of the competition, our model was ranked 3rd overall in BioLaySumm 2024 leaderboard.

heading: 2 Background

heading: 2.1 Task description
text: In BioLaySumm 2024, participants are expected to generate lay summaries for the research articles in the test set made from PLOS and eLife journals. For the development of summarization systems, PLOS (eLife) dataset provides 24773 (4346) articles for the train split and 1376 (241) articles for the validation split. For both PLOS and eLife datasets, the test split is composed of 142 articles. For each data point, the whole article including the abstract is provided along with the keywords and article id. For the train splits and the validation splits, groundtruth lay summaries targeted for non-experts are provided. These summaries are written by authors (PLOS) or expert editors (eLife). Participants can submit summaries generated from either individual models for each dataset or a unified model for both datasets. The qualities of submitted summaries are evaluated in three criteria: relevance, readability, and factuality. Each criterion is composed of multiple automatic metrics:
• Relevance: ROUGE (1,2, and L) (Lin, 2004), BERTScore (Zhang et al., 2020)
• Readability: Flesch-Kincaid Grade Level (FKGL) (Kincaid et al., 1975), Dale-Chall Readability Score (DCRS) (Dale and Chall, 1948), Coleman-Liau Index (CLI) (Coleman and Liau, 1975), LENS (Maddela et al., 2023)
• Factuality: AlignScore (Zha et al., 2023), SummaC (Laban et al., 2022)
These metrics are calculated through the BioLaySumm 2024 evaluation script1. For each metric, the average score over the entire prediction is reported. The goal of competition is to minimize FKGL, DCRS, and CLI and maximize all other metrics.

heading: 2.2 Related works
text: While automatic text summarization has long been the subject of interest for its wide applicability in various domains (El-Kassas et al., 2021; Allahyari et al., 2017), the advent of large language models (LLMs) has innovated the field drastically (Chang et al., 2024; Zhang et al., 2024; G et al., 2024).
As a subfield of text summarization, automatic lay summarization of biomedical literature obtained further attention for its close relationship with health literacy (Guo et al., 2021). Since most biomedical research articles assume readers are familiar with the scientific concepts and domainspecific languages of the field, it is important to measure and evaluate the readability of the generated summaries as well (Guo et al., 2021; Goldsack et al., 2022). On the other hand, fact-checking the lay summaries has been important as the use of LLMs becomes popular since LLMs are known to often experience hallucinations that generate misinformed texts (Zhang et al., 2023).
In this context, BioLaySumm provides a meaningful challenge where both the readability and factuality of summaries are evaluated (Goldsack et al., 2023, 2024). While various approaches were used for last year’s competition (Goldsack et al., 2023), the most successful approaches include few-shot prompting on GPT models (Turbitt et al., 2023), fine-tuning on FLAN-T5 models (Sim et al., 2023), and factorized energy-based model trained on BioBart model (Phan et al., 2023).

heading: 3 System overview
text: To find the best-performing system for BioLaySumm 2024, we experimented with several different systems based on the abstracts of the research articles. In this section, we introduce the systems we experimented including the system we submitted to the leaderboard of BioLaySumm 2024. Throughout all experiments, we used eLife (PLOS) training data only for model training or prompting
1https://github.com/TGoldsack1/BioLaySumm2024evaluation_scripts
to generate summaries for eLife (PLOS) validation/test data.

heading: 3.1 Submitted system: Unsupervised fine-tuned LoRA model
text: The system we submitted for the competition is the unsupervised fine-tuned LoRA model. Due to the context-size limitation of most LLMs, it is nearly impossible to fit the entire articles into the inputs for the LLMs. Instead, inspired by the system (Turbitt et al., 2023) which took 1st place in the last year’s competition (Goldsack et al., 2023), we only appended the abstract and the lay summary for the inputs to the model (Template 1). We used the entire input text for our training phase while we only used the input text just before the lay summary starts for the text generation. For parameterefficient training, we adopted low-rank adaptation (LoRA) (Hu et al., 2021) for our training.
### Provide a lay summary of the following research abstract.
Abstract: In temperate climates , winter deaths exceed summer ones . However , there is limited information on the timing and the relative magnitudes of maximum and minimum mortality , (...) Lay summary: In the USA , more deaths happen in the winter than the summer . But when deaths occur varies greatly by (...)
Template 1: Input text for unsupervised fine-tuning. The bold-faced text is the part used for the text generation as well.
While examining the generated summaries, we found that our fine-tuned model tends to repeat identical sentences rather than ending the summary. To regulate this, we post-processed our summary to chop off the redundant sentences. See appendix A For the details of the post-processing.

heading: 3.2 Other approaches

heading: 3.2.1 Baseline: zero-shot and few-shot prompting
text: While we use some form of fine-tuning in all the other approaches, we set a few-shot prompting system as our baseline following the best-performing system from the previous year’s competition (Turbitt et al., 2023). While we adopted this abstractbased few-shot approach from the last year’s competition, we randomly sampled 6 examples from the train set instead of hand-picked 3 examples used in the last year. We listed 6 abstract-summary pairs out of these sampled examples. See appendix B
for the sample prompt we used. Also, to provide a baseline that indicates the bare ability of the LLM we use, we tested zero-shot prompts where the same template was used as the few-shot prompts but with no examples listed.

heading: 3.2.2 Supervised fine-tuning with LoRA
text: Since the input text used in unsupervised finetuning in Section 3.1 trains not only the styles of lay summaries but also the styles of the original abstracts to the model, the quality of generated summaries may be affected by these abstracts in unwanted ways. To prevent this, we experimented with supervised fine-tuning. In particular, we treated the content of the lay summary as the label and the rest of the input text as the context by excluding input text tokens from the calculation of the loss function. To make this ‘label’ to be automatically detected after tokenization, we slightly changed the format of input text from Template 1 (see Appendix C).

heading: 3.2.3 Direct preference optimization on the fine-tuned model
text: Since our fine-tuning approaches only use abstractsummary pairs, it does not see the full contents of the research article during the training. Therefore the generated summaries may struggle with the factuality criterion. To mitigate this problem, we experimented with direct preference optimization (DPO) (Rafailov et al., 2024). DPO trains the human preference on a language model by providing pairs of similar samples where the relative preference within each pair is labeled (preferred sample vs rejected sample). To provide these relative preference labels, we generated summaries on randomly sampled 1000 articles in the train set using the unsupervised fine-tuned model and calculated factuality metrics (AlignScore, SummaC) on both the ground-truth lay summary and the generated summary. After comparing the average of the calculated factuality metrics within each pair, we label the summary with the higher score as the preferred sample and the summary with the lower score as the rejected sample. This DPO training is performed on top of the unsupervised fine-tuned model in Section 3.1.

heading: 4 Experimental setup

heading: 4.1 Hardware
text: All our experiments performed on a 4× Quadra RTX 8000 (48GB VRAM) card.

heading: 4.2 Text generation
text: We used mistral-7B-instruct-v0.2 throughout all experiments. For both the few-shot approach and the fine-tuned approach, text generation is performed through vLLM2 (Kwon et al., 2023) for faster experimentation. We set the temperature to 0 for all text generation.

heading: 4.3 Fine-tuning experiments with LoRA
text: For both unsupervised and supervised fine-tuning experiments, we utilized libraries from Huggingface (Transformers, PEFT3, TRL4). We used AdamW optimizer (Loshchilov and Hutter, 2017) to optimize cross-entropy loss with label smoothing (Pereyra et al., 2017). Experimented hyperparameters are available in Appendix D.

heading: 4.4 Direct preference optimization experiments
text: For DPO experiments, we utilized Axolotl library5. We used the sequence size of 4096, the batch size 8, and the learning rate 1.0 × 10−5 with a linear scheduler over 3 epochs.

heading: 5 Results

heading: 5.1 Experiment results
text: We report the results of all our experiments in Table 1. Averages of result 7 and result 8 are the scores submitted to the leaderboard of BioLaySumm2024, and our model is ranked 2nd in relevance, 16th in readability, 18th in factuality, and 3rd in average scores of all categories out of 55 participants (Goldsack et al., 2024). Overall, our model delivered decent summaries in all 3 evaluation criteria while particularly successful in the relevance criterion.

heading: 5.2 Analysis on approaches

heading: 5.2.1 Baseline approaches: zero-shot and few-shot prompting
text: We set the zero-shot and few-shot prompting system as our baseline following the most successful approach last year (Turbitt et al., 2023). When comparing the baseline results from others in Table 1 (result 1, 3 vs. result 5, 9∼13 and result 2, 4 vs. result 6), fine-tuning approaches outperform zero-shot or few-shot prompting in relevance. For readability, fine-tuning is superior for the eLife
2https://github.com/vllm-project/vllm 3https://github.com/huggingface/peft 4https://github.com/huggingface/trl 5https://github.com/OpenAccess-AI-Collective/axolotl
dataset (except for the LENS score) while the opposite is true for the PLOS dataset. This might be related to the worse readability of PLOS summaries that the authors write themselves. On the other hand, fine-tuning approaches yield higher factuality scores for the PLOS dataset while giving worse AlignScore and better SummaC scores for the eLife dataset. These contrastive patterns in readability and factuality among different datasets might indicate that readability and factuality are in a trade-off relationship, as simplified summaries may deliver less accurate information.

heading: 5.2.2 Unsupervised vs. supervised fine-tuning
text: By comparing the unsupervised fine-tuning experiments (results 5, 11) with the supervised finetuning experiments (results 9, 12) in Table 1, we find that unsupervised fine-tuning outperforms supervised fine-tuning in all metrics except CLI. Despite our expectation of supervised fine-tuning performing better in the readability scores from not learning the patterns in the abstracts, the supervised fine-tuning was not superior in the readability neither. Detailed investigations on the reasons for this difference between the supervised and the unsupervised fine-tuning would be a good subject for the future research.

heading: 5.2.3 Direct preference optimization
text: When comparing the results of DPO experiments (results 10, 13) with the results of their fine-tuned model before DPO training (results 5, 11) in Table 1, we observe that DPO training gives better factuality scores as expected, as well as improved readability scores except for LENS. Yet, DPO training makes relevance scores worse at the same time, as its training process suggests some ground truth summaries as rejected samples.

heading: 5.2.4 Post-processing
text: To investigate the effect of the post-processing unit, we evaluated predictions with no post-processing (results 11, 12, 13 in Table 1). The comparison with the results of post-processed summaries (results 5, 9, 10) shows that post-processed summaries are superior to non-processed summaries in both relevance and factuality. Regarding the readability, the effect of the post-processing unit is mixed, where the post-processing improves LENS while it worsens DCRS and CLI. For FKGL, the effect is not consistent over different experiments.

heading: 6 Conclusion
text: As we participated in BioLaySumm 2024, we experimented with different fine-tuning approaches with LoRA to generate summaries based on the given abstract of a biomedical research article. In particular, we explored unsupervised fine-
tuning, supervised fine-tuning, and direct preference optimization, and we concluded that our bestperforming model is the unsupervised fine-tuned model with post-processing to chop off repeated sentences. Our model achieved 3rd place overall in the leaderboard of BioLaySumm 2024. While our model was successful, it would be interesting to extend our approach to a variety of larger LLMs or to adopt other schemes to utilize the full article of the research paper instead of the abstracts. Potential future researches on analysis on different fine-tuning methodologies and benchmarking on evaluation criteria beyond the current challenge may deepen the understanding of our approach.

heading: 7 Limitations
text: Due to our limited resources, we only experimented with a single type of relatively small open-sourced model. Due to the limited context size of the model we used, our exploration of methods to utilize full research articles was limited to DPO which interacts with the full articles only through the factuality scores.
It is also worthwhile to mention that our approach was more successful in the relevance than other than two other evaluation criteria. This might be related with the fact that summaries more readable than the suggested golden summary might score less in the BERTScore. It would be interesting subject for the future researches to see how our approach performs in other summary evaluation criteria beyond the current challenge.

heading: A Details of post-processing
text: In the raw predictions of fine-tuned models, we observed that identical sentences are repeated without completing the paragraph in a small fraction of the generated summaries. To mitigate this, we introduced the post-processing unit to chop off the repeated sentences from the prediction. To do this, we first split the prediction into a sequence of sentences. Then we examine these sentences from the beginning of the sequence and drop the rest of the sequence when the given sentence has appeared before during the examination.
We split the prediction into sentences based on the appearance of sentence-ending punctuation marks like period (".") or question mark ("?"). Yet, there are some exceptions we had to handle in this process:
• If punctuation is in the middle of parentheses, does not end the sentence there.
• If a period is part of a URL address, which is specified by the beginning sequences ("www" or "http") and the ending sequences ("com", "edu", "gov", or "org"), then do not end the sentence at that period.
• If a period is part of commonly used abbreviations in academic writing ("et al .", "vs .", and "e . g ."), do not end the sentence at that period.
• If the previous word of a period is a single letter English alphabet, do not end the sentence there, since it is likely a part of a phrase for a subsection or abbreviation of names (ex: "a.1", "c. elegans", "George R. R. Martin").
• If a period is surrounded by Arabic numerals, do not end the sentence since it is likely a part of a floating number.

heading: B Few-shot prompt for the baseline system
text: ### Provide a lay summary of the following research abstract.
Abstract: The role of the cellular microenvironment in enabling metazoan tissue genesis remains obscure . Ctenophora has
recently emerged as (...) Lay summary: The emergence of the diversity of
multicellular animals involved cells joining together to form tissues and organs . The glue that (...)
Abstract: To evolve and to be maintained , seasonal migration , despite its risks , has to yield fitness benefits compared with
year-round residency . Empirical data supporting this (...)
Lay summary: Winter is one of the most challenging seasons for many animals . Cold temperatures , bad weather , short days , long nights and a shortage of food can impose (...)
Abstract: The adaptive prokaryotic immune system CRISPR-Cas provides RNA-mediated protection from invading genetic elements . The
fundamental basis of the system is (...) Lay summary: In most animals , the adaptive
immune system creates specialized cells that adapt to efficiently fight off any viruses or other pathogens that have invaded . (...)
Abstract: Adipose tissue is crucial for the maintenance of energy and metabolic homeostasis and its deregulation can lead to obesity and type II diabetes ( T2D ) .
(...) Lay summary: Obesity is a growing public health
concern around the world , and can lead to the development of type 2 diabetes , heart disease and cancer . (...)
Abstract: The roles played by cortical inhibitory neurons in experience-dependent plasticity are not well understood . Here we evaluate (...) Lay summary: What we see or fail to see through our eyes leaves a lasting impression by changing the strength of connections between (...)
Abstract: Numerous studies have established important roles for microRNAs ( miRNAs ) in regulating gene expression . Here , we report that miRNAs also serve as (...) Lay summary: To produce a protein from a gene , the sequence of the gene must be transcribed to produce a molecule of messenger RNA (
mRNA ) . (...)
Abstract: Midbrain dopamine neurons have been proposed to signal reward prediction errors as defined in temporal difference ( TD ) learning algorithms. (...) Lay summary:
Template 2: Sample few-shot prompt used for our baseline system. The 6 examples listed here are the actual examples we used for the eLife articles.
C Input text for supervised fine-tuning
### Provide a lay summary of the following research abstract.
### Abstract: In temperate climates , winter deaths exceed summer ones . However , there is limited information on the timing and the relative magnitudes of maximum and minimum mortality , (...)
### Lay summary: In the USA , more deaths happen in the winter than the summer . But when deaths occur varies greatly by (...)
Template 3: Input text for supervised fine-tuning. The bold-faced text is the context and the rest of the text is the label.

heading: D Fine-tuning hyperparameters

heading: Future Work
text: It would be interesting subject for the future researches to see how our approach performs in other summary evaluation criteria beyond the current challenge