Text Summarization on COVID-19 research data
By: Vamsi Krishna
Tags: Data Science, Python, Naturallanguageprocessing, Covid 19

Summarize massive text data into a short, precise, and fluent summary that help us consume relevant information faster.

Photo by Susan Yin on Unsplash

The entire world is currently experiencing COVID-19 virus. Ever wondered what researchers have been finding of this virus? White House, in collaboration with other leading research groups, prepared COVID-19 research data set for the global research community to help in understanding more about the disease by applying AI and Natural Language Processing Techniques(NLP). This dataset also includes publications on SARS-CoV-2 and other coronaviruses.

Going through each of the research paper from the dataset is not practically feasible as there are more than 130,000 articles and 60K+ of full texts. To overcome the challenge to go through each paper in detail, I have developed NLP based Text Summarization model that gives a summary of the text out of entire document so that reader can get a decent understanding of what article is about and what topic author had focused in the research paper.

Let us begin with the steps involved in the summarization of text from the corpus of the data, and then step by step to accomplish text summarization on COVID-19 dataset. One can use this approach on any text document to create shorter and meaningful summaries.

Text Summarization:

Text summarization is one of the statistical methods to produce summaries from vast chunks of texts. Text summarization can be achieved using Natural Language Processing techniques and using deep learning algorithms. This article covers the NLP technique for text summarization.

Five necessary steps for text summarization using NLP approach:

Tokenize the sentences in the text. Pre-process the text. Create sentence frequency matrix. Determine weighted matrix for each sentence. Rank the sentence based on highest weight.

Now, let’s jump into COVID-19 dataset and perform required steps for summarizing the text.

Import all necessary libraries

# Start with loading all necessary libraries

import re

import nltk

import heapq

import numpy as np

from os import path

from PIL import Image

from wordcloud import WordCloud, STOPWORDS, ImageColorGenerator

import pandas as pd

import matplotlib.pyplot as plt

% matplotlib inline

### load stopwords

nltk.download('stopwords')

Before we get into summarizing the text, let’s us create some descriptive analytics and distribution of the data on critical fields in the dataset.

### read the file

df_text = pd.read_csv(‘covid.csv’)

Create a list of custom keywords and add to the keywords from NLTK.

stopwords_custom = nltk.corpus.stopwords.words('english') customize_stop_words = ['common','review','describes','abstract','retrospective','chart','patients','study','may','associated','results','including','high' 'found','one','well','among','Abstract','provide','objective','objective:','background','range','features','participates','doi', 'preprint', 'copyright', 'org', 'https', 'et ','et' 'al', 'author', 'figure', 'table', 'rights', 'reserved', 'permission', 'use', 'used', 'using', 'biorxiv', 'medrxiv', 'license', 'fig', 'fig.', 'al.', 'Elsevier', 'PMC', 'CZI',



]

### append custom stopwords to default stopwords from NLTK

for i in customize_stop_words:

stopwords_custom.append(i)

Let’s see which words are frequently used in all 50K+ papers from the data set are. It looks like some of the most commonly used words are ‘respiratory’, ‘epidemiology’ which are relevant in the current COVID-19 situation.

Can we find any interesting critical words in publication titles from all the 50K+ research papers? Once again, there are some common words in larger text and the title of the papers.

From the below word cloud, we can also observe that some authors submitted multiple publications.

Also, ‘paper published year’ is one of the variables in the dataset. There are publications from the mid 20th century to till date. 2021 as Published year must be incorrect entry because this paper published in April 2020.

Distribution of publication year in entire dataset:

Pre-process the data:

Clean the text by replacing special characters, extra spaces etc..

def clean_text(text): '''Clean the text with special characters, extra spaces etc..'''

clean_abstract = []

clean_abstract = re.sub(r’\d’,’ ‘,text)

clean_abstract = re.sub(r’\W’, ‘ ‘,clean_abstract)

clean_abstract = re.sub(r’\s+’, ‘ ‘,text)

text = re.sub(r’\[[0–9]*\]’,’ ‘,text)

return text ### select any text from abstract for summarization'

text= str(df_text['abstract'][9999])

print(text)



Code in above cell performs pre-processing steps and generates clean text to feed into the model and full text after pre-processing shown below.

Tokenize and Remove Stop words from text.

Now, input the preprocessed text to next steps to exclude stopwords and tokenize the sentences. Code in the following cell also creates a frequency matrix for each sentence. Function(tokenise_text) returns tokenized sentences and frequency matrix

def tokenise_text(text):

'''remove stop words, tokenize the text and create word count with scores''' wordcnt = {} for word in nltk.word_tokenize(text):

if word not in stopwords_custom:

if word not in wordcnt.keys():

wordcnt[word] = 1

else:

wordcnt[word] += 1

## Tokenize the text

sentenses = nltk.sent_tokenize(text)

for key in wordcnt.keys():

wordcnt[key] = wordcnt[key]/max(wordcnt.values())

return wordcnt,sentenses wordcnt,sentenses = tokenise_text(text)

Create sentence frequency matrix, weighted sentence matrix and rank the sentence based on the weights.

Create a frequency matrix of all those words mapped to the number of times those words appeared in the whole paragraph and weighted sentence matrix as well. The weighted matrix created by all values corresponding to a word divided by the maximum value of among all words. Length of the desired sentence is one of the hyperparameters. One has to be mindful in choosing this because sometimes longer sentences are not essential but have high weights, and that may cause choosing the wrong sentence as top rank. Determining the number of best sentences as well is our choice. I went with picking the top 2 sentences as best sentences.

def summrize_text(wordcnt,sentenses):

'''summarize the text!! choose desired number of words in sentence and number of sentense'''

sent_score = {} for sentense in sentenses:

for word in nltk.word_tokenize(sentense.lower()):

if word in wordcnt.keys():

if len(sentense.split(‘ ‘)) < 25:

if sentense not in sent_score.keys():

sent_score[sentense] = wordcnt[word]

else:

sent_score[sentense] += wordcnt[word]



### Pick best sentence for summarized text

best_sent = heapq.nlargest(2,sent_score,key = sent_score.get)



return best_sent best_sent = summrize_text(wordcnt,sentenses)

Finally, print the summarized text for the text that I considered from the dataset.

for sentense in best_sent:

print(sentense)

As we can see, long text that we passed as input to the model, converted to a short and meaningful summary.

Here is the full code in readable format:

GitHub link to the code:

References:

Please comment or provide feedback :)