IITR at BioLaySumm Task 1:Lay Summarization of BioMedical articles using Transformers
Year: 2023

Abstract:
This paper presents our submission to the shared task-Lay Summarization of Biomedical Research Articles at BioNLP-2023 workshop(Goldsack et al., 2023). The purpose of this task is the summarize biomedical articles in a concise and less technical way increasing their readability and their reach to lay audiences. In this paper, we use BART-based summarization techniques. We used labels of sentences to improve the performance of our model. Our model achieved a rouge-1 score value of 42.89% and an FKGL score of 10.7901 in relevance and readability parameters respectively.

text: The 22nd Workshop on Biomedical Natural Language Processing and BioNLP Shared Tasks, pages 625–628 July 13, 2023 ©2023 Association for Computational Linguistics
This paper presents our submission to the shared task-Lay Summarization of Biomedical Research Articles at BioNLP-2023 workshop(Goldsack et al., 2023). The purpose of this task is the summarize biomedical articles in a concise and less technical way increasing their readability and their reach to lay audiences. In this paper, we use BART-based summarization techniques. We used labels of sentences to improve the performance of our model. Our model achieved a rouge-1 score value of 42.89% and an FKGL score of 10.7901 in relevance and readability parameters respectively.

heading: 1 Introduction
text: Scientific paper’s growth has increased a lot in recent times. Papers in various domains are used to share research data. So it is important to summarize these papers so as to decrease the workload of the researchers and also to reduce the gap between the public and researchers. Technical summary still is a difficult text to understand for non-research people and can lead to misinterpretation of information. In the context of Bio-medical articles the need for Lay summarization is higher because of the more dynamic and difficult terminology and also information misinterpretation having a direct impact on Human lives.
The shared task of BioNLP Lay Summarization of Biomedical Research Articles aims to improve the tools used for Lay summarization for training models which give realistic lay summaries. It has tasks in which data is to be trained on two datasets eLife and PLOS which are two large biomedical article datasets, both varying in sizes of summary and readability of summaries(Goldsack et al., 2022)(Luo et al., 2022). Our approach tries to build a model to keep the lay summary relevant to the original model and improve its readability.

heading: 2 Related Work
text: The models which were existing earlier are in general extractive models which include sentence selection or hybrid models selecting sentences and then summarizing them .(Cohan et al., 2018) was the first paper describing a model abstractive summarization.
LaySumm subtask of the CL-SciSumm 2020(Chandrasekaran et al., 2020) shared task series also had submissions that use abstractive summarization. The data insufficiency was found in the task as the sufficient number of annotated lay summaries were not present at the time of the task which will help in getting a more relevant and better lay summary. Many other models that are trained for Lay Summarization also faced the same problems due to a lack of data or data summaries being too large and not suitable for lay summary training.
Dimsum Lay summarization (Yu et al., 2020) used the CL-SciSumm dataset and a BART(Lewis et al., 2019) baseline along with sentence labels as external supervision signals, data augmentation and got better results of relevance and readability.
Corpora for lay summarization(Goldsack et al., 2022) introduced the dataset with the articles of eLife and PLOS each with annotated summaries to improve the existing literature of bio-medical articles available to train a lay summarization model.

heading: 3 Datasets
text: There were two datasets in the task eLife and PLOS. PLOS is the larger of the two datasets, containing 24, 773 instances for training and 1, 376 for validation (the same for each subtask). eLife contains 4, 346 instances for training and 241 for validation. Each dataset contains train, validation, and test datasets in the form of JSON files.
Each sample contains articles with the following structure:
625
• Id-denoting the id of the article.
• Year-year in which article is published.
• Title-title of the article.
• Sections-Various sections of article which include:
– Introduction- Gives the context and initial idea of the article.
– Results and methods-These sections discuss the methods used in the article and their results.
– Discussions-These sections discuss the inferences from the results and reasons for the observations.
– Abstract-This section highlights the important points in the article.
– Summary-Contains the annotated lay summary of the article.
• Headings-Contains information about all the headings of the article.
• Keywords-Contains the keywords of the article.
The eLife has an average summary length of around 300-350 words whereas the summary length of PLOS is way shorter i.e. around 160-200 words. The PLOS dataset summary is also closer to its abstract whereas eLife summaries are a more simplified version of text i.e. more readable.(Goldsack et al., 2022)
The table 1 shows that the most significant contributions of the lay summary in PLOS and eLife come from the Introduction, and abstract sections.

heading: 4 Data Pre-processing
text: Datasets as mentioned were in the JSON format with various headings as keys of JSON dictionary objects. These datasets are converted into a CSV
file with each heading as a column. The irrelevant information columns like title, id, year are removed from the CSV file. Also as per table 1, sections like results and discussion don’t contribute to the final lay summary. So even those columns are removed from CSV files. The information thus present in the CSV file is only an introduction, abstract, and summary.
A new text column that contains a fraction(k) of the Introduction and its concatenation with the abstract is made which is helpful to train the model with Introduction+abstract as the input text. K is a hyper-parameter and it was found out by experimentation that the optimal value was around 0.6. The text is truncated to a max length of 1024 tokens due to the capacity of the BART model.

heading: 5 System Overview

heading: 5.1 Baseline
text: We used the Facebook BART-based model as our baseline.BART(Lewis et al., 2019) is a transformer encoder-decoder (seq2seq) model with a bidirectional (BERT-like) encoder and an auto-regressive (GPT-like) decoder (Vaswani et al., 2017).
BART is very effective when it is fine-tuned with domain-specific datasets for text-generation and summarization tasks. BART is pre-trained by corrupting text with an arbitrary noising function and learning a model to reconstruct the original text. We used BART fine-tuned on the CNN/DailyMail dataset (Hermann et al., 2015) as baseline model.

heading: 5.2 Fine Tuning
text: We have fine-tuned the baseline with both eLife and PLOS datasets. We used the abstract+Fraction of Introduction as our input and summary as our output. We have performed training for around 25000 iterations and chose checkpoints with the least validation loss. We also evaluated the BART model which is not fine-tuned on the datasets to get a base score.

heading: 5.3 Metrics
text: We have used the Rouge1, Rouge2, and RougeL values for evaluating the relevance of our summary with respect to the actual article and we have used FKGL, and DCRS scores for measuring the readability of our summaries.

heading: 5.4 Libraries and Language
text: Our experiments are conducted on a supercomputer environment with the Conda 3.7 module environment and Python 3.7.2 language. Also, Pandas = 1.3.5 and transformers = 4.28.1 are used.

heading: 5.5 Testing
text: Testing is done by taking article sections as input to the fine-tuned model and summary sections as reference summaries to calculate the rouge scores. During testing due to the large size of article data and 1024 token capacity of the BART model summarizer input we had to give the input in the form of chunks i.e the article is divided into chunks of less than 1024 tokens and then pass the input. The maxlength and min-length parameters of the summarizer are adjusted so that the output summary length is in the range of the average summary length of the dataset (different for PLOS and eLife).

heading: 6 Experiments
text: We did experiments with different parameters as follows:
• BART(Abstract-eLife): BART is chosen as the baseline model and the abstract of the eLife article is taken as input to the BART model.
• BART(Introduction-eLife): BART is chosen as the baseline model and the Introduction of the eLife article is taken as input to the BART model.
• BART(Introduction+abstract-eLife): BART is chosen as the baseline and part of the Introduction along with the abstract is given as input.
• T5(Introduction+abstract-eLife): T5 is chosen as the baseline and part of the Introduction along with the abstract is given as input.
For the hyper-parameters, we have considered a batch size of 4 due to the memory constraints of GPU. We have performed 15 epochs and chose a model with the least validation loss. We have taken the implementation of BART from hugging-face transformers.1

heading: 7 Results
text: We have calculated the Rouge1, Rouge2, and RougeL values for all the experiments. We can see that BART baseline without any fine-tuning gives Rouge1 value of around 27% showing that without fine-tuning the summarization might be more technical and fine-tuning is required for summarization to be lay.
We can see from the table that using the data of the sections of Introduction and abstract combined with BART baseline gives better results than using any of them separately probably due to Introduction not having relevant key points of the article and abstract size in eLife being very small compared to the summary.
Also, t5 results are a bit less accurate than the BART model showing BART model performance when fine-tuned with proper domain-related datasets gives very good results Also, the PLOS dataset tested with the eLife model has almost the same rouge values as the PLOS model so considering the PLOS model has less readability as compared to the eLife model we considered the eLife model for both the datasets for the final submission.
Ranking Phase: We have submitted the BART model fine-tuned with the eLife dataset. Our scores for relevance and readability are 42.81% for Rouge1, 32.7% for Rouge2, 10.7 for FKGL score, and 8.85 for DCRS. The ranks obtained for relevance and readability are 17 and 1 respectively.

heading: 8 Conclusion
text: Our results show that the BART model fine-tuned gives the best results as compared with other models. Also, the PLOS dataset has an abstract near to
1https://github.com/huggingface/transformers
its summary so using the eLife model for summarization gives more readability.
Limitations
We can improve our model in the future by using the keywords given in the dataset to enhance the quality of the summaries generated by using keyword embedding in addition to the fine-tuned model.

heading: Future Work
text: Limitations
We can improve our model in the future by using the keywords given in the dataset to enhance the quality of the summaries generated by using keyword embedding in addition to the fine-tuned model