Keyword Extraction — A Benchmark of 7 Algorithms in Python | by Andrea D'Agostino | Towards Data Science

I’ve been actively working on finding a valid algorithm for a keyword extraction task. The goal was to find an algorithm capable of extracting keywords in an efficient way, balancing quality of the extraction and time of execution, as my corpus of data was increasing hitting millions of rows quickly. One of the KPIs was to extract keywords that always made sense alone, by themselves, even if out of context.
This led me to test and experiment with several well-known keyword extraction mechanisms. Here’s me sharing my small journey with you.
I have used the following libraries to conduct the study
NLTK, to help me in the preprocessing phases and for some helper functions
RAKE
YAKE
PKE
KeyBERT
Spacy
Pandas and Matplotlib, together with other generic but core libraries were used as well.
The way the benchmark works is the following
We’ll first import the dataset that contains our textual data. We’ll then create separate functions that apply the extraction logic
__algorithm_name__(str: text) → [keyword1, keyword2, ..., keywordn]
Then we’ll create a function that applies the extraction on the entire corpus.
extract_keywords_from_corpus(algorithm, corpus) → {algorithm, corpus_keywords, elapsed_time}
Spacy will then help us define a matcher object that will return true or false if a keyword matches a syntactic pattern that makes sense for our task
Finally we’ll bundle up everything in a function that outputs our final report.
I am working on a series of small chunks of text taken from the internet. This is a sample
['To follow up from my previous questions. . Here is the result!\n', 'European mead competitions?\nI’d love some feedback on my mead, but entering the Mazer Cup isn’t an option for me, since shipping alcohol to the USA from Europe is illegal. (I know I probably wouldn’t get caught/prosecuted, but any kind of official record of an issue could screw up my upcoming citizenship application and I’m not willing to risk that).\n\nAre there any European mead comps out there? Or at least large beer comps that accept entries in the mead categories and are likely to have experienced mead judges?', 'Orange Rosemary Booch\n', 'Well folks, finally happened. Went on vacation and came home to mold.\n', 'I’m opening a gelato shop in London on Friday so we’ve been up non-stop practicing flavors - here’s one of our most recent attempts!\n', "Does anyone have resources for creating shelf stable hot sauce? Ferment and then water or pressure can?\nI have dozens of fresh peppers I want to use to make hot sauce, but the eventual goal is to customize a recipe and send it to my buddies across the States. I believe canning would be the best way to do this, but I'm not finding a lot of details on it. Any advice?", 'what is the practical difference between a wine filter and a water filter?\nwondering if you could use either', 'What is the best custard base?\nDoes someone have a recipe that tastes similar to Culver’s frozen custard?', 'Mold?\n'
Mostly are food-related items. We’ll take a sample of 2000 documents to test our algorithms.
We will not preprocess our texts just yet, because some of the algorithms base their results on presence of stopwords and punctuation.
Let’s define the keyword extraction functions.
Each extractor takes in as an argument the text from which we want to extract keywords and returns a list of keywords, from the best to the worse according to their weighing technique. Pretty straightforward.
Note: for some reason, I couldn’t initialize all extractor objects outside the functions. TopicRank and MultiPartiteRank threw errors whenever I did that. Performance-wise this isn’t perfect, but the benchmark can be done nonetheless.
We are already restricting some of the accepted grammar patterns by passing pos = {‘NOUN’, ‘PROPN’, ‘ADJ’, ‘ADV’} — this, together with Spacy, will ensure that almost all the keywords will be sensical from a human language perspective. We also want keywords to be at least trigrams, just to have more specific keywords and avoid going too general. Check the libraries documentation to go deeper into the paramters and how they work.
Now let’s define a function that will apply a single extractor to the entire corpus while outputting some information too.
All this function does is populate a dictionary with the data coming from the extractor passed in as argument and a series of useful information like how much time it took to execute the task.
This is where we make sure that the keywords that are returned by the extractors always (almost?) make sense. For instance,
We can clearly understand that the first three keywords can live on their own. They have a meaning, they are completely sensical. When we are is not — we require more information to understand the meaning of that chunk of data. We want to avoid this.
Spacy comes in handy with the Matcher object. We’ll define a match function that takes in a keyword and returns True or False if the defined patterns match.
We are almost done. This is the last step before launching the script and gathering the results.
We’ll define a benchmark function that takes in our corpus and a boolean for shuffling or not our data. For each extractor, it calls the extract_keywords_from_corpus function, which returns a dictionary containing the result of that extractor. We store that value in a list.
For each algorithm in the list, we compute
average number of extracted keywords
average number of matched keywords
compute a score that takes into account the average number of matches found divided by how much time it took to perform the operation
We store all of our data in a Pandas DataFrame and we export it to .csv.
To run the benchmark is easy as writing
And here are the results
and a bar chart with the performance score
Rake wins on all other algorithms by a great deal according to the score formula which is (avg_matched_keywords_per_document/ time_elapsed_in_seconds). The fact that Rake processes 2000 documents in 2 seconds is impressive, and even though the accuracy isn’t as high as Yake or KeyBERT, the time factor makes it win over the others.
If we were to consider only accuracy, computed as the ratio between avg_matched_keywords_per_document and avg_keywords_per_document, we get these results
Rake is performing quite well also from the accuracy perspective. It makes sense to have such a high performance score given the short amount of time it takes to perform the extraction.
If we didn’t have time in the equation, KeyBERT would definitely take the winning spot as the most accurate algorithm capable of extracting sensical keywords.
The aim of this project was to find the best algorithm in terms of efficiency. For this task, Rake seems to take that spot.
Bottom line, if you require accuracy over anything else, KeyBERT is your solution, otherwise Rake or Yake. I would use Yake in the cases I have no particular goals and just want a balanced solution.
Campos, R., Mangaravite, V., Pasquali, A., Jatowt, A., Jorge, A., Nunes, C. and Jatowt, A. (2020). YAKE! Keyword Extraction from Single Documents using Multiple Local Features. In Information Sciences Journal. Elsevier, Vol 509, pp 257–289. pdf
Campos R., Mangaravite V., Pasquali A., Jorge A.M., Nunes C., and Jatowt A. (2018). A Text Feature Based Automatic Keyword Extraction Method for Single Documents. In: Pasi G., Piwowarski B., Azzopardi L., Hanbury A. (eds). Advances in Information Retrieval. ECIR 2018 (Grenoble, France. March 26–29). Lecture Notes in Computer Science, vol 10772, pp. 684–691. pdf
Campos R., Mangaravite V., Pasquali A., Jorge A.M., Nunes C., and Jatowt A. (2018). YAKE! Collection-independent Automatic Keyword Extractor. In: Pasi G., Piwowarski B., Azzopardi L., Hanbury A. (eds). Advances in Information Retrieval. ECIR 2018 (Grenoble, France. March 26–29). Lecture Notes in Computer Science, vol 10772, pp. 806–810.
Csurfer. (n.d.). CSURFER/Rake-nltk: Python implementation of the rapid automatic keyword extraction algorithm using NLTK. Retrieved November 25, 2021, from https://github.com/csurfer/rake-nltk
Liaad. (n.d.). Liaad/Yake: Single-document unsupervised keyword extraction. Retrieved November 25, 2021, from https://github.com/LIAAD/yake
Boudinfl. (n.d.). BOUDINFL/pke: Python keyphrase extraction module. Retrieved November 25, 2021, from https://github.com/boudinfl/pke
MaartenGr. (n.d.). MAARTENGR/Keybert: Minimal keyword extraction with bert. Retrieved November 25, 2021, from https://github.com/MaartenGr/KeyBERT
Explosion. (n.d.). Explosion/spacy: 💫 industrial-strength natural language processing (NLP) in Python. Retrieved November 25, 2021, from https://github.com/explosion/spaCy