Extracting Keyphrases from Text: RAKE and Gensim in Python | by Nikita Saxena | Towards Data Science

The Washington Post says it publishes an average of 1,200 stories, graphics, and videos per day (that count, though, includes wire stories). That’s a lot of content! Who has the time to go through all of that news? Won’t it be awesome if we could extract just the relevant phrases from every news article?
Keyphrases are a set of words (or groups of words) that summarize the important points of the paragraph. It isn’t a conclusive summary of the text, it’s just a list of relevant concepts that were discussed in the article.
Lemmatize Text: It doesn’t make sense to include each and every word in the vocabulary of the text passage when words like writing’, ‘written’, ‘wrote’ as they mean the same: ‘write’. So, we lemmatize text, i.e., bring each word to its root form before anything else.Select Potential Phrases: Text passages contain many words, but not all of them are relevant. Most of them might be frequently used words like ‘a’, ‘that’, ‘then’ and so on. Such words, called stopwords, must be filtered else they will contaminate the output. Consecutive words bearing contextual similarity must be grouped together.Score Each Phrase: Once you have a list of possible phrases, you need to rank them to figure out which one is the most important.
Lemmatize Text: It doesn’t make sense to include each and every word in the vocabulary of the text passage when words like writing’, ‘written’, ‘wrote’ as they mean the same: ‘write’. So, we lemmatize text, i.e., bring each word to its root form before anything else.
Select Potential Phrases: Text passages contain many words, but not all of them are relevant. Most of them might be frequently used words like ‘a’, ‘that’, ‘then’ and so on. Such words, called stopwords, must be filtered else they will contaminate the output. Consecutive words bearing contextual similarity must be grouped together.
Score Each Phrase: Once you have a list of possible phrases, you need to rank them to figure out which one is the most important.
In this blog, we will explore python implementations of the following keyphrase extraction algorithms:
RAKE
RAKE-NLTK
Gensim
Let’s take a sample text passage so that we can compare outputs for all algorithms:
AI Platform Pipelines has two major parts: (1) the infrastructure for deploying and running structured AI workflows that are integrated with Google Cloud Platform services and (2) the pipeline tools for building, debugging, and sharing pipelines and components. The service runs on a Google Kubernetes cluster that’s automatically created as a part of the installation process, and it’s accessible via the Cloud AI Platform dashboard. With AI Platform Pipelines, developers specify a pipeline using the Kubeflow Pipelines software development kit (SDK), or by customizing the TensorFlow Extended (TFX) Pipeline template with the TFX SDK. This SDK compiles the pipeline and submits it to the Pipelines REST API server, which stores and schedules the pipeline for execution.
RAKE stands for Rapid Automatic Keyword Extraction. The algorithm itself is described in the Text Mining Applications and Theory book by Michael W. Berry. We’ll just go through the implementation here, I’d recommend this site in case you wanna learn what goes behind the scenes.
Installation:
Imports:
Stopwords:
Download this list of stopwords from here and save the path in a variable called stop_dir.
Create a RAKE Object from Text:
Now, it’s time to extract the keywords! RAKE doesn’t originally print keywords in order of score. But it returns the score and the extracted keyphrases. Let’s write a quick function to sort these extracted keyphrases and scores.
Store the text passage in a variable and pass it to the rake_object. We named our variable subtitles. The 10 at the end of first-line implies we are extracting only the top 10 keywords.
Bingo! We’re done! The higher the score, the important the keyphrase.
keywords: [(‘installation process’, 4.0), (‘tensorflow extended’, 4.0), (‘sharing pipelines’, 5.4), (‘google kubernetes cluster’, 9.5), (‘ai platform pipelines’, 10.4), (‘google cloud platform services’, 15.0), (‘cloud ai platform dashboard’, 15.0), (‘pipelines rest api server’, 15.4), (‘running structured ai workflows’, 15.5), (‘kubeflow pipelines software development kit’, 23.4)]
RAKE-NLTK is a modified version that uses the natural language processing toolkit NLTK for some of the calculations.
Installation:
Import, Declare a RAKE-NLTK Object and Extract! We again extract just the top 10 keywords.
Here’s the output for the same text passage using RAKE-NLTK. For the chosen passage, RAKE and RAKE-NLTK give the same output. But this isn’t always the case. Try it out for yourself on other passages!
[‘kubeflow pipelines software development kit’, ‘running structured ai workflows’, ‘pipelines rest api server’, ‘cloud ai platform dashboard’, ‘google cloud platform services’, ‘ai platform pipelines’, ‘google kubernetes cluster’, ‘two major parts’, ‘sharing pipelines’, ‘tensorflow extended’]
Implemented in Gensim Python and Cython, Gensim is an open-source library for natural language processing, using modern statistical machine learning.
Installation:
Imports and the Function Call:
Output:
pipelines pipeline platform developers development sdk tfx kubernetes
It seems like a simple keywords function call in Gensim doesn't perform inbuilt preprocessing.
We learned how to write Python codes to extract keywords from text passages.
Here are some other cool keyphrase extraction implementations. Check them out!
NLTK
TextRank
You could try sample text passages on all these algorithms and see what suits your use case best!
Came across a different keyphrase extraction algorithm? Drop it in the comments!