Topic Modelling in Python with spaCy and Gensim | by Tarek Ghanoum | Towards Data Science

Hackers recently attacked an international company. They copied several thousand documents and published the data on the dark web. The company reached out to my employer and asked if we could go through the data to assess the type of information contained in the documents. My first thought was: Topic Modelling.
Topic Modelling is a technique to extract hidden topics from large volumes of text. The technique I will be introducing is categorized as an unsupervised machine learning algorithm. The algorithm's name is Latent Dirichlet Allocation (LDA) and is part of Python's Gensim package.
LDA was first developed by Blei et al. in 2003. LDA is a generative probabilistic model similar to Naive Bayes. It represents topics as word probabilities and allows for uncovering latent or hidden topics as it clusters the words based on their co-occurrence in a respective document.
Using the most advanced tools on the market (i.e. PowerPoint), I created the image below depicting the overall process. I will start the process by collecting the documents (Step 1); afterwards, I will do some data cleaning and break down all the documents into tokens (Step 2). From the tokens, I can build a dictionary that gives each token a unique ID number, which can then be used to create a corpus or Bag of Words representing the frequency of the tokens (Step 3). I use my dictionary and corpus to build a range of topics and try to find the optimal number of topics (Step 4). The last step is to find the distribution of topics in each document (Step 5).
Before we dive into the above-visualized steps, I will ask you to go through the code below and ensure everything is installed and imported.
!pip install pyLDAvis -qq!pip install -qq -U gensim!pip install spacy -qq!pip install matplotlib -qq!pip install seaborn -qq!python -m spacy download en_core_web_md -qqimport pandas as pdimport matplotlib.pyplot as pltimport seaborn as snssns.set()import spacyimport pyLDAvis.gensim_modelspyLDAvis.enable_notebook()# Visualise inside a notebookimport en_core_web_mdfrom gensim.corpora.dictionary import Dictionaryfrom gensim.models import LdaMulticorefrom gensim.models import CoherenceModel
The data I will be using contains an overview of 500 different reports, including their summaries. I have made the data available on my GitHub so feel free to follow along or download the data and edit the URL.
reports = pd.read_csv('https://github.com/sg-tarek/Python/raw/main/cordis-h2020reports.gz')reports.head()
The snippet above only shows the first six columns, although the data contains a total of fifteen columns.
reports.info()
In the field of Natural Language Processing (NLP), text preprocessing is the practice of cleaning and preparing text data. I will be using an open-source software library called spaCy to prepare the data for analysis, but other libraries such as NLTK can also be used.
I told you earlier to import and download something called 'en_core_web_md' which is spaCy's pre-trained model. The model, which I will call 'nlp', can be thought of as a pipeline. When you call 'nlp' on a text or word, the text runs through a processing pipeline, which is depicted below. It means that if the text isn't tokenized, it will then be tokenized, and afterwards, different components (tagger, parser, ner etc.) will be activated. To tokenize text means turning a string or document into smaller chunks (tokens).
The model is specially trained on English text (notice the 'en' in the model name), making it capable of detecting different English words. Other language models are also supported.
The most interesting component in the pipeline is the tagger which assigns Part-Of-Speech (POS) tags based on SpaCy's English language model to gain a variety of annotations. A POS tag (or part-of-speech tag) is a special label assigned to each token in a text corpus to indicate the type of token (is it an adjective? Punctuation? A verb? etc.) and often also other grammatical categories such as tense, number (plural/singular), symbols etc. POS tags are used in corpus searches and in-text analysis tools and algorithms.
Some of the POS tags are listed below:
We can use the POS tags to preprocess the data by removing unwanted tags. Assuming we want to remove all numbers in our text, we can then point at a specific tag and remove it.
I will focus on the 'summary' column, where I will tokenize, lemmatize and remove stopwords:
# Our spaCy model:nlp = en_core_web_md.load()# Tags I want to remove from the textremoval= ['ADV','PRON','CCONJ','PUNCT','PART','DET','ADP','SPACE', 'NUM', 'SYM']tokens = []for summary in nlp.pipe(reports['summary']):   proj_tok = [token.lemma_.lower() for token in summary if token.pos_ not in removal and not token.is_stop and token.is_alpha]   tokens.append(proj_tok)
Notes to expand your knowledge:
You might notice the 'is_alpha' and 'is_stop', which are attributes connected to specific tokens; you can look at the full list here: https://spacy.io/api/token
Another point is 'nlp.pipe', which is specifically used to process text as a sequence of strings. This is much more efficient than processing text one by one. If you're only processing a single text, simply remove the '.pipe' extension. Source: https://spacy.io/usage/processing-pipelines
Let's bring the tokens back into the report by creating a new column:
reports[‘tokens’] = tokensreports[‘tokens’]
The two main inputs to the LDA topic model are the dictionary and the corpus:
Dictionary: The idea of the dictionary is to give each token a unique ID.
Corpus: Having assigned a unique ID to each token, the corpus simply contains each ID and its frequency (if you wanna dive into it, then search for Bag of Word (BoW) which will introduce you to word embedding).
# I will apply the Dictionary Object from Gensim, which maps each word to their unique ID:dictionary = Dictionary(reports['tokens'])
You can print the dictionary which will tell you that 8848 unique ID’s were found. We can look at some of the IDs assigned to the tokens:
print(dictionary.token2id){‘available’: 0, ‘capability’: 1, ‘cea’: 2, ‘challenge’: 3, ‘chemistry’: 4, ‘chimie’: 5, ‘coating’: 6, ‘commercial’: 7, ‘company’: 8, ‘compliant’: 9, ‘concentrated’: 10, ‘conductive’: 11, ‘conductivity’: 12, ‘cost’: 13, ‘demand’: 14, ‘design’: 15, ‘dispersion’: 16, ‘easily’: 17,...
I will filter out low-frequency and high-frequency tokens, also limit the vocabulary to a max of 1000 words:
dictionary.filter_extremes(no_below=5, no_above=0.5, keep_n=1000)
No_below: Tokens that appear in less than 5 documents are filtered out.
No_above: Tokens that appear in more than 50% of the total corpus are also removed as default.
Keep_n: We limit ourselves to the top 1000 most frequent tokens (default is 100.000). Set to ‘None’ if you want to keep all.
We are now ready to construct the corpus using the dictionary from above and the doc2bow function. The function doc2bow() simply counts the number of occurrences of each distinct word, converts the word to its integer word id and returns the result as a sparse vector:
corpus = [dictionary.doc2bow(doc) for doc in reports['tokens']]
The next step is to train the unsupervised machine learning model on the data. I choose to work with the LdaMulticore, which uses all CPU cores to parallelize and speed up model training. If this doesn’t work for you for some reason, try the gensim.models.ldamodel.LdaModel class which is an equivalent, but more straightforward and single-core implementation.
When inserting our corpus into the topic modelling algorithm, the corpus gets analyzed in order to find the distribution of words in each topic and the distribution of topics in each document.
lda_model = LdaMulticore(corpus=corpus, id2word=dictionary, iterations=50, num_topics=10, workers = 4, passes=10)
As input, I give the model our corpus and dictionary from before; besides, I choose to iterate over the corpus 50 times to optimize the model parameters (this is the default value). I select the number of topics to be ten and the workers to be 4 (find the number of cores on your PC by pressing the ctr+shift+esc keys). The pass is 10, which means the model will pass through the corpus ten times during training.
Having trained the model, the next natural step is to evaluate it. After having constructed the topics, a coherence score can be computed. The score measures the degree of semantic similarity between high scoring words in each topic. In this fashion, a coherence score can be computed for each iteration by inserting a varying number of topics.
A range of algorithms has been introduced to calculate the coherence score (C_v, C_p, C_uci, C_umass, C_npmi, C_a, ...). Working with the gensim library makes computing these coherence measures for topic models fairly simple. I personally choose to implement C_v and C_umass. The coherence score for C_v ranges from 0 (complete incoherence) to 1 (complete coherence). Values above 0.5 are fairly good, according to John McLevey (source: Doing Computational Social Science: A Practical Introduction By John McLevey). On the other hand, C_umass returns negative values.
Below I simply iterate through a different number of topics and save the coherence score in a list. Afterwards, I plot using seaborn.
Calculating the coherence score using C_umass:
topics = []score = []for i in range(1,20,1):   lda_model = LdaMulticore(corpus=corpus, id2word=dictionary, iterations=10, num_topics=i, workers = 4, passes=10, random_state=100)   cm = CoherenceModel(model=lda_model, corpus=corpus, dictionary=dictionary, coherence='u_mass')   topics.append(i)   score.append(cm.get_coherence())_=plt.plot(topics, score)_=plt.xlabel('Number of Topics')_=plt.ylabel('Coherence Score')plt.show()
Calculating the coherence score using C_v:
topics = []score = []for i in range(1,20,1):   lda_model = LdaMulticore(corpus=corpus, id2word=dictionary, iterations=10, num_topics=i, workers = 4, passes=10, random_state=100)   cm = CoherenceModel(model=lda_model, texts = reports['tokens'], corpus=corpus, dictionary=dictionary, coherence='c_v')   topics.append(i)   score.append(cm.get_coherence())_=plt.plot(topics, score)_=plt.xlabel('Number of Topics')_=plt.ylabel('Coherence Score')plt.show()
When looking at the coherence using the C_umass or C_v algorithm, the best is usually the max. Looking at the graphs I choose to go with 5 topics, although no certain answer can be given.
lda_model = LdaMulticore(corpus=corpus, id2word=dictionary, iterations=100, num_topics=5, workers = 4, passes=100)
We can now print out the five topics and the related words:
lda_model.print_topics(-1)
[(0, ‘0.015*”datum” + 0.013*”service” + 0.010*”research” + 0.010*”support” + 0.009*”european” + 0.009*”network” + 0.009*”social” + 0.009*”people” + 0.008*”provide” + 0.008*”platform”’),
(1, ‘0.021*”disease” + 0.016*”patient” + 0.015*”health” + 0.015*”clinical” + 0.012*”treatment” + 0.012*”study” + 0.010*”drug” + 0.010*”cancer” + 0.009*”system” + 0.009*”care”’),
(2, ‘0.021*”system” + 0.015*”technology” + 0.013*”market” + 0.013*”energy” + 0.012*”high” + 0.012*”cost” + 0.010*”product” + 0.010*”process” + 0.009*”base” + 0.009*”solution”’),
(3, ‘0.034*”cell” + 0.013*”study” + 0.013*”plant” + 0.012*”new” + 0.012*”understand” + 0.010*”mechanism” + 0.009*”process” + 0.009*”human” + 0.008*”development” + 0.008*”aim”’),
(4, ‘0.026*”research” + 0.024*”innovation” + 0.016*”sme” + 0.015*”market” + 0.013*”new” + 0.012*”business” + 0.011*”plan” + 0.011*”partner” + 0.011*”support” + 0.010*”development”’)]
Let’s look at our first report summary:
reports['summary'][0]“Polyaniline has historically been one of the most promising conductive polymers from a cost/performance perspective, but processing issues have limited its uptake. Building upon seminal work on polyaniline performed at the CEA, RESCOLL (an independent research company based in France specialized in chemistry materials) has developed and patented a new electrically conductive polyaniline formulation under the trade name of PANIPLASTTM....”
According to our LDA model, the above text belongs to Topic 2 and 4. The article is 87% belonging to topic 2 (index 1) and 12% belonging to topic 4 (index 3).
lda_model[corpus][0][(2, 0.870234), (4, 0.122931786)]
Let’s Visualize the topics and the words in each topic. Just like we saw above, but notice that the numbers in the circles do not align with the numbers from ‘print_topics’ above:
lda_display = pyLDAvis.gensim_models.prepare(lda_model, corpus, dictionary)pyLDAvis.display(lda_display)
The above chart represents our five topics as circles. They have been drawn using a dimensionality reduction technique called PCA. The goal is to have a distance in order to avoid overlapping and make each circle unique. When I hover over a circle different words are displayed on the right, showing word frequency (blue) and estimated term frequency within the selected topic (red). Topics closer to each other are more related.
We can create a new column in the dataframe which has the most probable topic that each article belong to. We can add the most probable topic by running through each summary text:
reports['topic'] = [sorted(lda_model[corpus][text])[0][0] for text in range(len(reports['summary']))]
Let’s count the frequency of each topic:
reports.topic.value_counts()
The count shows:
Topic 1: 320
Topic 2: 86
Topic 3: 57
Topic 4: 19
Topic 5: 18
This aligns well with the visualization from before.
We can share our topic visualization by uploading it to GitHub pages. Click here to see my topics and interact with them.
The first step is to download the visualization we created earlier in an HTML format. It’s essential to keep the name as index.html since the GitHub pages will be looking for a file with this name:
pyLDAvis.save_html(lda_display, ‘index.html’)
You will need to edit the index file to make it work properly. The file contains a lot of HTML code, regardless of the content copy the lines below and add them to your file. Notice that some code is at the top, while </body> and </html> goes at the bottom:
<!DOCTYPE html>   <html lang="en">        <body>LDA CODE GOES HERE!  </body>  </html>
The next step is to create a new repository on GitHub and use the following naming convention for the repo:
<username>.github.io
Afterwards, drop your index.html file in the repo and visit the Topic Modelling visualization on the same name as the repo: <username>.github.io.
I hope you enjoyed this article as much as I have enjoyed writing it. Leave a comment if you have any difficulties understanding my code. The Data Science community has given me a lot, so I am always open to giving back.
Feel free to connect with me on Linkedin and follow me on Medium to receive more articles.