A Better Approach to Text Summarization | by Vinicius Monteiro | Towards Data Science

There are several approaches to perform automatic text summarization. It can be done with supervised or unsupervised learning, with deep or only machine learning. And inside these categories, there are a wide variety of methods.
In terms of types of summaries, there are two — extractive and abstractive. The program presented here uses unsupervised learning and generates an extractive summarization.
Extractive summarization is when the summary is a subset of the original text because all words in the summary are included in the original text. For an overview of the automatic text summarization field, I recommend this survey.
In general, in a simplistic explanation, a summarizer’s goal is to find the most relevant words and then select the sentences that contain those words to be part of the summary.
What about words that are different but have the same meaning?
For instance,
Harry moved to the US.
The prince relocated to North America.
They have no words in common, but the two sentences are strongly related. “The” and “to” doesn’t count. These words would’ve have been removed in the preprocessing step. More about it below.
In a naive approach, “Harry” or “US” could be selected as relevant words, but the sentence “The prince relocated to North America” would have no chance to be part of the summary.
The technique presented in this research paper resolves this problem.
The code shown in this post is only a slightly modified version (structure, method name, etc.) of the source available on Github.
The text I use in this tutorial is the same one used in this excellent article by Praveen Dubey. Make sure to read it as well (I changed one of the sentences for proof of concept).
The summarizer relies on the word embeddings, so it also selects sentences that contain words with the same meaning as the most relevant words (centroid) — even if the words are different.
For more details on word embeddings, check out this article by Zhi Li
So, before even preprocessing the text, the program must define the embedding model. I used the text being summarized as the data to get the embedding model.
But it could be the Wikipedia dump or some other text. There are already available models as well that can be used. To obtain the word embedding, I split the text into words and passed it to Word2vec.
for sent in clean_sentences:    words.append(nlkt_word_tokenize(sent))model = Word2Vec(words, min_count=1, sg = 1)
You must preprocess the text. It includes splitting the text into sentences, lowering the case of all words, removing stopwords (is, an, the, etc.) and punctuation and other tasks.
I covered this step alone in another article. The goal is not to waste resources (compute power, time) processing things that don’t add much value to extracting the semantics and understanding the text.
Especifically to identifying and splitting the text into sentences, it’s crucial as later the algorithm will score and select them to be part of the summary.
def sent_tokenize(text):    sents = nlkt_sent_tokenize(text)    sents_filtered = []    for s in sents:        sents_filtered.append(s)    return sents_filtereddef cleanup_sentences(text):    stop_words = set(stopwords.words('english'))    sentences = sent_tokenize(text)    sentences_cleaned = []    for sent in sentences:        words = nlkt_word_tokenize(sent)        words = [w for w in words if w not in string.punctuation]        words = [w for w in words if not w.lower() in stop_words]        words = [w.lower() for w in words]        sentences_cleaned.append(" ".join(words))    return sentences_cleaned
The algorithm uses TF-IDF to find the most relevant words of the text. These words are the centroid of the text.
After finding the words centroid, the program sums up the vectors of the words that are part of the centroid, and that sum is the embedding representation of the centroid.
Let’s say the most relevant words are: Microsoft, program, AI
The embedding representation of the centroid (most relevant terms) = Microsoft vector + program vector + AI vector.
To remind, the embedding model represents each word as a vector. That’s how you have a vector for each word.
def build_embedding_representation(words, word_vectors, embedding_model):    embedding_representation = np.zeros(embedding_model.vector_size, dtype="float32")    word_vectors_keys = set(word_vectors.keys())    count = 0    for w in words:        if w in word_vectors_keys:            embedding_representation = embedding_representation + word_vectors[w]            count += 1    if count != 0:       embedding_representation = np.divide(embedding_representation, count)    return embedding_representationdef get_tf_idf(sentences):    vectorizer = CountVectorizer()    sent_word_matrix = vectorizer.fit_transform(sentences)transformer = TfidfTransformer(norm=None, sublinear_tf=False, smooth_idf=False)    tfidf = transformer.fit_transform(sent_word_matrix)    tfidf = tfidf.toarray()centroid_vector = tfidf.sum(0)    centroid_vector = np.divide(centroid_vector, centroid_vector.max())feature_names = vectorizer.get_feature_names()relevant_vector_indices = np.where(centroid_vector > 0.3)[0]word_list = list(np.array(feature_names)[relevant_vector_indices])    return word_listcentroid_words = get_tf_idf(clean_sentences)centroid_vector = build_embedding_representation(centroid_words, word_vectors, emdedding_model)
The sentences are scored based on how similar they are to the centroid embedding.
In order to compare to the centroid embedding, the algorithm calculates the embedding representation of each sentence.
Sentence embedding = sum of words vectors that are part of the sentence.
For instance,
Sentence1 = word1, word2, word3Sentence1 embedding = word1 vector + word2 vector + word3 vector
Lastly, after the sentence embedding is defined, the algorithm uses cosine similarity to calculate the similarity between the centroid and the sentence embedding.
Each sentence gets a score based on how similar they are to the centroid.
def build_embedding_representation(words, word_vectors, embedding_model):    embedding_representation = np.zeros(embedding_model.vector_size, dtype="float32")    word_vectors_keys = set(word_vectors.keys())    count = 0    for w in words:        if w in word_vectors_keys:            embedding_representation = embedding_representation + word_vectors[w]            count += 1    if count != 0:       embedding_representation = np.divide(embedding_representation, count)    return embedding_representationsentences_scores = []for i in range(len(clean_sentences)):        scores = []        words = clean_sentences[i].split()        #Sentence embedding representation        sentence_vector = build_embedding_representation(words,    word_vectors, emdedding_model)        #Cosine similarity between sentence embedding and centroid embedding        score = similarity(sentence_vector, centroid_vector)        sentences_scores.append((i, raw_sentences[i], score, sentence_vector))sentence_scores_sort = sorted(sentences_scores, key=lambda el: el[2], reverse=True)sentence_scores_sort = sorted(sentences_scores, key=lambda el: el[2], reverse=True)
Sentences score
The sentence in red has no words part of the centroid. Still, it had a good score. Higher than another one that includes centroid words.
The sentences are selected based on their score. The number of sentences selected is limited by how many words the summary should contain (50 words, 100 words, or?).
A common problem when dealing with automatic summarization is handling redundancy — too similar sentences being part of the summary.
To overcome that, when selecting the sentences, you compare them with the sentences already in summary. If a chosen sentence is too similar to one in summary, you don’t add it to the final text.
The algorithm uses cosine similarity with some predefined threshold when making the comparison.
count = 0sentences_summary = []#Handle redundancyfor s in sentence_scores_sort:        if count > 100:            break        include_flag = True        for ps in sentences_summary:            sim = similarity(s[3], ps[3])            if sim > 0.95:                include_flag = False        if include_flag:            sentences_summary.append(s)            count += len(s[1].split())sentences_summary = sorted(sentences_summary, key=lambda el: el[0], reverse=False)summary = "\n".join([s[1] for s in sentences_summary])
Original Text
Generated summary
The code was slightly modified/simplified for me to understand better and explain in this article. But you can view the research paper author’s Github for the complete and more accurate implementation.
I find this project really interesting. It’s more efficient than other algorithms that relies on bag of words (BOW) or TF-IDF only. The research paper I used as my main reference is much more detailed. It evaluates the generated summary against other algorithms and covers multi-language documents summarization.
Evaluation comes next. I plan to write another post covering ROUGE, a tool commonly used to score how good the summary is. I’ll also dive into summarizers based on supervised machine learning. And deep learning as well.
Thanks for reading.
[1] Text Summarization Techniques: A Brief Survey. By Mehdi Allahyari, Seyedamin Pouriyeh, Mehdi Assefi, Saeid Safaei, Elizabeth D. Trippe, Juan B. Gutierrez, Krys Kochut
[2] Centroid-based Text Summarization through Compositionality of Word Embeddings. By Gaetano Rossiello, Pierpaolo Basile and Giovanni Semeraro | Github page: https://github.com/gaetangate/text-summarizer