NLP: Building Text Summarizer — Part 1
By: Ashish Singhal
Tags: Machine Learning, Data Science, NLP, Text Mining, Text Summarization

As the model consists quite a few modules and to keep this article short, this article will cover Text Pre-processing and Tokenization modules.

The extractive model’s core lies within TF-IDF and Cosine Similarities modules. Before applying these two on the text document read by the model, its need to be processed and ready in the format required by the TF-IDF and Cosine Similarities.

1. Text Preprocessing

After model reading the text document, the text document goes to preprocessing module which does the followings in the sequence mentioned below:

1. Lower case the text

2. Removing special characters

3. Removing numbers

4. Tokenize the text

5. Remove the stop words

6. Remove the words which have length equal to 1 after removing special characters and numbers.

Lower case the text to avoid any ambiguity between the words having same meaning but different case and leave mid-sentence words as they are because if they’re starting with upper case somewhere inside the sentence, that means it is a name or an entity and that word have more importance than other words. As the word has more importance hence the parent sentence has a greater importance and so it should and might be present in the summary text.

After removing special characters and numbers and tokenizing the text, there might be words which only have one character. These words would not have any meaning; and hence they are removed.

Before preprocessing the text, sentences are tokenized and stored. This is needed to generate the final summary based on the sentence index we get to put in the summary.

file = 'input.txt'

file = open(file, 'r')

text = file.read() tokenizedSentence = sent_tokenize(text)

text = preprocessText(text)

tokenizedWords = tokenizing(text) def preprocessText(text)

processedText = remove_special_characters(str(text))

processedText = re.sub(r'\d+', '', processedText)

return processedText def tokenizing(text)

tokenized_words_with_stopwords = word_tokenize(text)

tokenized_words = [word for word in tokenized_words_with_stopwords if word not in stopWords]

tokenized_words = [word for word in tokenized_words if len(word) > 1]

tokenized_words = [word.lower() for word in tokenized_words]

return tokenized_words

2. Generating N-Grams

This module takes processed tokenized words from first step and a number ’n’ as the inputs and gives n-grams as its output.

N-grams have been used to group the words resulting into phrases which would give more insights about the sentence. These insights will help in assigning the importance score to each phrase and hence we can get the importance score for parent sentence too. These importance score will help in extracting sentences for summary text.

nGramNumber = 2 tokenizedWordsWithNGrams = generate_ngrams(tokenizedWords, nGramNumber) def generate_ngrams(listOfWords, n):

nGramsList = []

for num in range(0, len(listOfWords)):

nGramWord = ' '.join(listOfWords[num:num+n])

nGramsList.append(nGramWord)

return nGramsList

Until now, the extraction model has read the text document, preprocessed it and have generated N-grams. Along with it, model also has sentences in tokenized format. These two things would be serve to TF-IDF module followed by Cosine Similarity module.

TF-IDF and Cosine Similarity module is explained in Part 2.