Summarize Long Text Documents using Machine Learning | by Satyam Kumar | Towards Data Science

Text summarization is the process of creating a short, accurate, and fluent summary of a long text document. It is the process of distilling the most important information for a text document. The intention of text summarization is to create a summary of a large corpus having important points describing the entire corpus.
A vast quantity of text data is generated on the internet, be it social media articles, news articles, etc. Manual creation of summaries is time-consuming, and therefore a need for automatic text summaries has arisen. Not only are the automatic summarization tools much faster, but they are also less biased than humans.
Text summarization is useful in many ways to summarize the textual data. Some of the uses of text summarization are:
To reduce the reading time of long documents
While researching documents, summaries make the selection process easier.
The text summarizes are useful for question-answer systems.
Using a text summarization system enables commercial abstract services to increase the number of text documents to process. Several news portals such as Google News, Inshorts, etc provide short summaries of the long news article for their readers.
Sumy is an open-sourced Python library to extract summaries from HTML pages and text files. The package also contains an evaluation framework for text summaries. Sumy offers several algorithms and methods for text summarization, some of them are:
Luhn: heuristic method
Latent Semantic Analysis (LSA)
Edmundson heuristic method
LexRank
TextRank
and many more. Visit the Sumy documentation page to know more about the text summarization algorithm Sumy offers.
In this article, we will discuss and implement some of the popular text summarization algorithms Luhn, LexRank, LSA.
Sumy is an open-sourced Python library and can be installed using PyPl:
pip install sumy
To read text documents from string variable or file, we can use a PlainTextParser from the Sumy package.
To read text document from a text string variable ‘document’:
parser = PlaintextParser.from_string(document, Tokenizer("english"))
To read text document from a text file with location as ‘file’:
parser = PlaintextParser.from_file(file, Tokenizer("english"))
After reading the text document in Sumy Text Parser, we can use several algorithms or methods to summarize the given text document.
LexRank Algorithm is an unsupervised approach to summarization and is inspired by the PageRank algorithm. LexRank uses an IDF-modified cosine similarity score to improve the Pagerank score for document summarization. It summarizes the text based on graph-based centrality scoring of sentences.
If one sentence is very similar to other sentences in the text corpus, then that sentence is considered of great importance. Such sentences can be recommended to the users.
LexRank summarizer can be implemented using an object called from Sumy package. After creating an object of LexRankSummarizer, pass the text document and number of lines to return in the summary.
Latent Semantic Analysis (LSA) is based on term frequency techniques with singular value decomposition to summarize texts. LSA is an unsupervised NLP technique, and the aim of LSA is to create a representation of text data in terms of topics or latent features.
LSA consists of two steps:
To generate a document term matrix (or numerical vector).
To perform Singular Value Decomposition on document term matrix. SVD reduces the dimensionality of the original dataset by encoding it with latent features.
With LSA these latent features represent topics in the original data.
Text summarizers are used to create short summaries of long text documents. The input long text documents are taken from the Wikipedia page of Machine Learning.
The output of Text Summarization using LexRank Algorithm:
The output of Text Summarization using LSA Algorithm:
In this article, we have discussed how to summarize long text documents using an open-sourced python package Sumy. Sumy offers various algorithms such as LexRank, Luhn, LSA, TextRank, SumBasic, etc to summarize the text documents. Sumy can summarize texts from string variables and files on disk.
You can also read about Text-Summarize, that is another open-source python library to summarize text document.
[1] Sumy Documentation (May 19, 2019): https://pypi.org/project/sumy/
Thank You for Reading