Summarize whole paragraph to one sentence by Extractive Approach | by Edward Ma | Towards Data Science

To catch a quick idea of long document, we will always to do a summarization when we read a article or book. In English, the first (or first two) sentence(s) of each article has a very high chance of representing the whole article. Of course, the topic sentence can be the last sentence in sometimes.
In NLP, there are two approaches to do the text summarization. The first one, extractive approach, is a simple approach which is extracting key words or sentences from article. There are some limitations and proved that the performance is not very good. The second one, abstractive approach, is generating a new sentences base on given article. It needs more advanced technique.
After read this article:
Understand PageRank algorithmUnderstand TextRank algorithmHow can we use TextRank algorithm to have a summarization
Understand PageRank algorithm
Understand TextRank algorithm
How can we use TextRank algorithm to have a summarization
PageRank algorithm is developed by Google for searching the most importance of website so that Google search result is relevant to query.
In PageRank, it is a directed graph. At the beginning, all node have equal score (1 / total number of node).
The algorithm
The first formula is the simplified version of PageRank and we will use this one for demo. The second one is a little bit complicated as it involved one more parameter which is damping factor, “d”. By default d is 0.85
Let take a look in the simplified version. In iteration 1, here is how PageRank calculate:
A: (1/4)/3. As only C is pointing to A, so we use previous C score (iteration 0) divided by number of node (i.e. 3) that C is pointing
B: (1/4)/2 + (1/4)/3. Both A and C are pointing to B, so previous A score (iteration 0) divided by number of node (i.e. 2) that A is pointing. For C, it is same as previous one which is (1/4)/3.
For detail, you may checkout the video for full explantation.
Question: When should we stop the iteration?
According to theory, it should calculate until no big update on score.
Why we need to introduce PageRank before TextRank? Because the idea of TextRank comes from PageRank and using similar algorithm (graph concept) to calculate the importance.
Difference:
TextRank graph is undirected. Meaning that all edge are bidirectionalThe weight of edge is difference while it is 1 in PageRank. There are different way to calculate such as BM25, TF-IDF.
TextRank graph is undirected. Meaning that all edge are bidirectional
The weight of edge is difference while it is 1 in PageRank. There are different way to calculate such as BM25, TF-IDF.
There are a lot of different document similarity implementation such as BM25, cosine similarity, IDF-modified-cosine. You may choose the best fit for your problem. If you do not have idea about those algorithm, please let us know and we will include it in later sharing.
gensim provides a simple API to calculate TextRank by using BM25 (Best Match 25).
Step 1: Environment Setup
pip install gensim==3.4.0
Step 2: Import library
import gensim print('gensim Version: %s' % (gensim.__version__))
Result:
gensim Version: 3.4.0
Step 3: Initial testing content
# Capture from https://www.cnbc.com/2018/06/01/microsoft--github-acquisition-talks-resume.htmlcontent = "Microsoft held talks in the past few weeks " + \    "to acquire software developer platform GitHub, Business " + \    "Insider reports. One person familiar with the discussions " + \    "between the companies told CNBC that they had been " + \    "considering a joint marketing partnership valued around " + \    "$35 million, and that those discussions had progressed to " + \    "a possible investment or outright acquisition. It is " + \    "unclear whether talks are still ongoing, but this " + \    "person said that GitHub's price for a full acquisition " + \    "was more than Microsoft currently wanted to pay. GitHub " + \    "was last valued at $2 billion in its last funding round " + \    "2015, but the price tag for an acquisition could be $5 " + \    "billion or more, based on a price that was floated " + \    "last year. GitHub's tools have become essential to " + \    "software developers, who use it to store code, " + \    "keep track of updates and discuss issues. The privately " + \    "held company has more than 23 million individual users in " + \    "more than 1.5 million organizations. It was on track to " + \    "book more than $200 million in subscription revenue, " + \    "including more than $110 million from companies using its " + \    "enterprise product, GitHub told CNBC last fall.Microsoft " + \    "has reportedly flirted with buying GitHub in the past, " + \    "including in 2016, although GitHub denied those " + \    "reports. A partnership would give Microsoft another " + \    "connection point to the developers it needs to court to " + \    "build applications on its various platforms, including " + \    "the Azure cloud. Microsoft could also use data from " + \    "GitHub to improve its artificial intelligence " + \    "producs. The talks come amid GitHub's struggle to " + \    "replace CEO and founder Chris Wanstrath, who stepped " + \    "down 10 months ago. Business Insider reported that " + \    "Microsoft exec Nat Friedman -- who previously " + \    "ran Xamarin, a developer tools start-up that Microsoft " + \    "acquired in 2016 -- may take that CEO role. Google's " + \    "senior VP of ads and commerce, Sridhar Ramaswamy, has " + \    "also been in discussions for the job, says the report. " + \    "Microsoft declined to comment on the report. " + \    "GitHub did not immediately return a request for comment."
Step 4: Try different ratio
Ratio parameter use for decide number of import sentences are returned.
Original Content:Microsoft held talks in the past few weeks to acquire software developer platform GitHub, Business Insider reports. One person familiar with the discussions between the companies told CNBC that they had been considering a joint marketing partnership valued around $35 million, and that those discussions had progressed to a possible investment or outright acquisition. It is unclear whether talks are still ongoing, but this person said that GitHub's price for a full acquisition was more than Microsoft currently wanted to pay. GitHub was last valued at $2 billion in its last funding round 2015, but the price tag for an acquisition could be $5 billion or more, based on a price that was floated last year. GitHub's tools have become essential to software developers, who use it to store code, keep track of updates and discuss issues. The privately held company has more than 23 million individual users in more than 1.5 million organizations. It was on track to book more than $200 million in subscription revenue, including more than $110 million from companies using its enterprise product, GitHub told CNBC last fall.Microsoft has reportedly flirted with buying GitHub in the past, including in 2016, although GitHub denied those reports. A partnership would give Microsoft another connection point to the developers it needs to court to build applications on its various platforms, including the Azure cloud. Microsoft could also use data from GitHub to improve its artificial intelligence producs. The talks come amid GitHub's struggle to replace CEO and founder Chris Wanstrath, who stepped down 10 months ago. Business Insider reported that Microsoft exec Nat Friedman -- who previously ran Xamarin, a developer tools start-up that Microsoft acquired in 2016 -- may take that CEO role. Google's senior VP of ads and commerce, Sridhar Ramaswamy, has also been in discussions for the job, says the report. Microsoft declined to comment on the report. GitHub did not immediately return a request for comment.---> Summarized Content (Ratio is 0.3):On Friday, Business Insider reported that Microsoft has held talks to buy GitHub — a $2 billion startup that claims 24 million software developers as users.Just to catch you up: GitHub is an online service that allows developers to host their software projects.---> Summarized Content (Ratio is 0.5):On Friday, Business Insider reported that Microsoft has held talks to buy GitHub — a $2 billion startup that claims 24 million software developers as users.Microsoft declined to comment, but you can read the full Business Insider report here.While we wait for further word on the future of GitHub, one thing is very clear: It would make perfect sense for Microsoft to buy the startup.Just to catch you up: GitHub is an online service that allows developers to host their software projects.---> Summarized Content (Ratio is 0.7):On Friday, Business Insider reported that Microsoft has held talks to buy GitHub — a $2 billion startup that claims 24 million software developers as users.Microsoft declined to comment, but you can read the full Business Insider report here.While we wait for further word on the future of GitHub, one thing is very clear: It would make perfect sense for Microsoft to buy the startup.Just to catch you up: GitHub is an online service that allows developers to host their software projects.From there, anyone from all over the world can download those projects and submit their own improvements.development world.
Result
Original Content:On Friday, Business Insider reported that Microsoft has held talks to buy GitHub — a $2 billion startup that claims 24 million software developers as users. It's not immediately clear what will come of these talks. Microsoft declined to comment, but you can read the full Business Insider report here. While we wait for further word on the future of GitHub, one thing is very clear: It would make perfect sense for Microsoft to buy the startup. If the stars align, and GitHub is integrated intelligently into Microsoft's products, it could give the company a big edge against Amazon Web Services, the leading player in the fast-growing cloud market. Just to catch you up: GitHub is an online service that allows developers to host their software projects. From there, anyone from all over the world can download those projects and submit their own improvements. That functionality has made GitHub the center of the open source software. development world.---> Summarized Content (Ratio is 0.3):On Friday, Business Insider reported that Microsoft has held talks to buy GitHub — a $2 billion startup that claims 24 million software developers as users.Just to catch you up: GitHub is an online service that allows developers to host their software projects.---> Summarized Content (Ratio is 0.5):On Friday, Business Insider reported that Microsoft has held talks to buy GitHub — a $2 billion startup that claims 24 million software developers as users.Microsoft declined to comment, but you can read the full Business Insider report here.While we wait for further word on the future of GitHub, one thing is very clear: It would make perfect sense for Microsoft to buy the startup.Just to catch you up: GitHub is an online service that allows developers to host their software projects.---> Summarized Content (Ratio is 0.7):On Friday, Business Insider reported that Microsoft has held talks to buy GitHub — a $2 billion startup that claims 24 million software developers as users.Microsoft declined to comment, but you can read the full Business Insider report here.While we wait for further word on the future of GitHub, one thing is very clear: It would make perfect sense for Microsoft to buy the startup.Just to catch you up: GitHub is an online service that allows developers to host their software projects.From there, anyone from all over the world can download those projects and submit their own improvements.development world.
Step 5: Try different word count
Word count parameter is another variable to control the result. If both word_count and ratio are entered. Ratio will be ignored.
print('Original Content:')print(content)for word_count in [10, 30, 50]:    summarized_content = gensim.summarization.summarize(body, word_count=word_count)    print()    print('---> Summarized Content (Word Count is %d):' % word_count)    print(summarized_content)
Result
riginal Content:Microsoft held talks in the past few weeks to acquire software developer platform GitHub, Business Insider reports. One person familiar with the discussions between the companies told CNBC that they had been considering a joint marketing partnership valued around $35 million, and that those discussions had progressed to a possible investment or outright acquisition. It is unclear whether talks are still ongoing, but this person said that GitHub's price for a full acquisition was more than Microsoft currently wanted to pay. GitHub was last valued at $2 billion in its last funding round 2015, but the price tag for an acquisition could be $5 billion or more, based on a price that was floated last year. GitHub's tools have become essential to software developers, who use it to store code, keep track of updates and discuss issues. The privately held company has more than 23 million individual users in more than 1.5 million organizations. It was on track to book more than $200 million in subscription revenue, including more than $110 million from companies using its enterprise product, GitHub told CNBC last fall.Microsoft has reportedly flirted with buying GitHub in the past, including in 2016, although GitHub denied those reports. A partnership would give Microsoft another connection point to the developers it needs to court to build applications on its various platforms, including the Azure cloud. Microsoft could also use data from GitHub to improve its artificial intelligence producs. The talks come amid GitHub's struggle to replace CEO and founder Chris Wanstrath, who stepped down 10 months ago. Business Insider reported that Microsoft exec Nat Friedman -- who previously ran Xamarin, a developer tools start-up that Microsoft acquired in 2016 -- may take that CEO role. Google's senior VP of ads and commerce, Sridhar Ramaswamy, has also been in discussions for the job, says the report. Microsoft declined to comment on the report. GitHub did not immediately return a request for comment.---> Summarized Content (Word Count is 10):---> Summarized Content (Word Count is 30):On Friday, Business Insider reported that Microsoft has held talks to buy GitHub — a $2 billion startup that claims 24 million software developers as users.---> Summarized Content (Word Count is 50):On Friday, Business Insider reported that Microsoft has held talks to buy GitHub — a $2 billion startup that claims 24 million software developers as users.Just to catch you up: GitHub is an online service that allows developers to host their software projects.
For entire code, you may check out from github. Let us know if you also want to understand about abstractive approach. Will arrange an article later on
According to gensim source code, at least 10 sentences is recommend for the input
No training data or model building is required.
It fits not only English but also any other a bag of input (Symbol, Japanese etc). You may also read TextRank research paper for detail understanding.
From my experience, the result is not good in most of the time. It may due to variety of words and the result is only a subset of input.
I am Data Scientist in Bay Area. Focusing on state-of-the-art in Data Science, Artificial Intelligence , especially in NLP and platform related.
Visit my blog from ttp://medium.com/@makcedward/
Get connection from https://www.linkedin.com/in/edwardma1026
Explore my code from https://github.com/makcedward
Check my kernal from https://www.kaggle.com/makcedward