Short Text Topic Modeling. Intuition and (some) maths to... | by Matyas Amrouche | Towards Data Science

Topic Modeling aims to find the topics (or clusters) inside a corpus of texts (like mails or news articles), without knowing those topics at first. Here lies the real power of Topic Modeling, you don’t need any labeled or annotated data, only raw texts, and from this chaos Topic Modeling algorithms will find the topics your texts are about!
In this post we will describe the intuition and logic behind the most popular approach for Topic Modeling, the LDA, and see its limitation on short texts. Given this post is about Short Text Topic Modeling (STTM) we will not dive into the details of LDA. The reader willing to deepen his knowledge of LDA can find great articles and useful resources about LDA here and here. Then, in a second part, we will present a new approach for STTM and finally see in a third part how to easily apply it (fit/predict ✌️) on a toy dataset and evaluate its performance.
The reader already familiar with LDA and Topic Modeling may want to skip the first part and directly go to the second and third ones which present a new approach for Short Text Topic Modeling and its Python coding 🐍.
The most popular Topic Modeling algorithm is LDA, Latent Dirichlet Allocation. Let’s first unravel this imposing name to have an intuition of what it does.
Latent because the topics are “hidden”. We have a bunch of texts and we want the algorithm to put them into clusters that will make sense to us. For example, if our text data come from news content, typically the clusters found might be about Mideast Politics, Computer, Space... but we don’t know it yet.
Dirichlet stands for the Dirichlet distribution the model uses as a prior to generate document-topic and word-topic distributions.
Allocation because we want to allocate topics to our texts.
Figure 1 below describes how the LDA steps articulate to find the topics within a corpus of documents.
“A document is generated by sampling a mixture of these topics and then sampling words from that mixture” (Andrew Ng, David Blei and Michael Jordan from the LDA original paper).
NB: In the Figure 1 above, we have set K=3 topics and N=8 words in our vocabulary for illustration ease. We also named these topics Computer, Space and Mideast Politics for illustration ease (rather than calling them topic 1, topic 2 and topic 3). Indeed, it will be our task to understand that the 3 found topics are about Computer, Space and Mideast Politics regarding their content (we will see this part more in depth during the topic attribution of our STTM pipeline in part III).
In short, LDA by using Dirichlet distributions as prior knowledge generates documents made of topics and then update them until they match the ground truth.
Despite its great results on medium or large sized texts (>50 words), typically mails and news articles are about this size range, LDA poorly performs on short texts like Tweets, Reddit posts or StackOverflow titles’ questions.
Looking at the short texts examples above on Figure 2, it is evident that the assumption that a text is a mixture of topics (remember first step in Figure 1) is not true anymore. We will now assume that a short text is made from only one topic.
The Gibbs Sampling Dirichlet Mixture Model (GSDMM) is an “altered” LDA algorithm, showing great results on STTM tasks, that makes the initial assumption: 1 topic ↔️1 document. The words within a document are generated using the same unique topic, and not from a mixture of topics as it was in the original LDA.
Before diving into code and practical aspects, let’s understand GSDMM with an equivalent procedure called the Movie Group Process that will help us understand the different steps and process under the hood of STTM, and how to tune efficiently its hyper-parameters (we remember alpha and beta from the LDA part).
Imagine a bunch of students in a restaurant, seating randomly at K tables. They are all asked to write their favorite movies on a paper (but it must remain a short list). The objective is to cluster them in such a way that so students within the same group share the same movie interest. To do so, one after another, students must make a new table choice regarding the two following rules:
Rule 1: Choose a table with more students. This rule improves completeness, all students sharing the same movie’s interest are assigned to the same table.
Rule 2: Choose a table where students share similar movie’s interest. This rule aims to increase homogeneity, we want only members sharing the same movie’s interest at a table.
After repeating this process, we expect some tables to disappear and others to grow larger and eventually have clusters of students matching their movie’s interest. This is simply what the GSDMM algorithm does!
In this part we will build full STTM pipeline from a concrete example using the 20 News Groups dataset from Scikit-learn used for Topic Modeling on texts.
First thing first, we need to download the STTM script from Github into our project folder.
cd sttm_projectgit clone https://github.com/rwalk/gsdmm.git
Now, we can start implementing the STTM pipeline (here is a static version of the notebook I used).
# Useful libsfrom sklearn.datasets import fetch_20newsgroupsimport pickleimport pandas as pdimport numpy as np# STTM lib from Githubfrom gsdmm import MovieGroupProcess# Custom python scripts for preprocessing, prediction and# visualization that I will define more in depth laterfrom preprocessing import tokenizefrom topic_allocation import top_words, topic_attributionfrom visualisation import plot_topic_notebook# Load the 20NewsGroups dataset from sklearncats = ['talk.politics.mideast', 'comp.windows.x', 'sci.space']newsgroups_train = fetch_20newsgroups(subset=’train’, categories=cats)
However, in this exercise, we will not use the whole content of the news to extrapolate a topic from it, but only consider the Subject and the first sentence of the news (see Figure 3 below). Indeed, we need short texts for short texts topic modeling... obviously 🙏
Besides, we will only look at only 3 topics (evenly distributed among the dataset), for illustration ease. These topics are the following:
Mideast Politics 🌍
Space 👾
Windows X 🖥
# Preprocessing and tokenizationtokenized_data = tokenize(df, form_reduction='stemming', predict=False)
Here are the preprocessing recipe I have followed for this task:
Tokenization using spaCy tokenizer.
Removing stop words and 1 character words.
Stemming (given my empirical experience I have observed that stemming gives better clusters on short text compared to lemmatization) using the nltk library’s stemmer.
Removing empty documents and documents with more than 30 tokens.
Removing unique token (with a term frequency = 1).
However, one must keep in mind that preprocessing is data dependent and should consider to adapt an other preprocessing approach if a different dataset is used.
Now that our data are cleaned and processed to the proper input format, we are ready to train the model 🚀
# Train STTM model# Init of the Gibbs Sampling Dirichlet Mixture Model algorithm# K = number of potential topic (which we don't know a priori)# alpha = # beta = # n_iters = number of iterations to mgp = MovieGroupProcess(K=10, alpha=0.1, beta=0.1, n_iters=30)vocab = set(x for doc in docs for x in doc)n_terms = len(vocab)y = mgp.fit(docs, n_terms)# Save modelwith open(‘dumps/trained_models/v1.model’, “wb”) as f: pickle.dump(mgp, f) f.close()
Let’s dive under the hood and understand the hyper-parameters machinery of the GSDMM model 🛠:
K = 10. In a real case we are not aware of the exact number of topic so we want to choose a higher value. Theoretically, GSDMM should empty useless clusters and eventually find the exact number of cluster. It will not be the case here, but nothing to worry, we will explain the situation more in depth later.
alpha = 0.1 and beta = 0.1. Here we kept the default parameters (which work well for several datasets). However, one might want to tune them to improve its topic allocation regarding the completeness and homogeneity of the clusters. Don’t hesitate to refer to the original paper 📖 to understand the balance between these two parameters.
n_iters = 30. According to the original paper, GSDMM converges quite fast (about 5 iterations) on several datasets and remain very stable. Hence, 30 iterations is a good default value for any kind of dataset.
Once the model is trained, we want to explore the topics found and check if they are coherent regarding their content 🔎
Given that our model has gathered the documents into 10 topics, we must give them a name that will make sense regarding their content. So let’s dive into the topics found by our model.
doc_count = np.array(mgp.cluster_doc_count)print('Number of documents per topic :', doc_count)print('*'*20)# Topics sorted by the number of document they are allocated totop_index = doc_count.argsort()[-10:][::-1]print('Most important clusters (by number of docs inside):', top_index)print('*'*20)# Show the top 5 words in term frequency for each cluster top_words(mgp.cluster_word_distribution, top_index, 5)
The code above display the following statistics that give us insight about what our clusters are made of.
Ideally, the GSDMM algorithm should find the correct number of topics, here 3, not 10. Three explanations come to my mind:
Find other hyper-parameters to empty smaller cluster (refer to original paper for a deeper understanding of alpha and beta parameters).
We have both small dataset and vocabulary (about 1700 documents and 2100 words), which may be difficult for the model to extrapolate and distinguish significant difference between topics. As usual, the more data, the better.
The algorithm might found topics inside the topics. Let me explain. As we well know, one of the topic is about Mideast news. However, the algorithm split this topic into 3 sub-topics: tension between Israel and Hezbollah (cluster 7), tension between Turkish government and Armenia (cluster 5) or Zionism in Israel (cluster 0).
However, even if there are more than 3 found clusters, it’s pretty obvious how we can assign them to their respective general topic.
# Must be hand made so the topic names match the above clusters # (Figure 5) regarding their contenttopic_dict = {}topic_names = ['x',               'mideast',               'x',               'space',               'space',               'mideast',               'space',               'space',               'mideast',               'space']for i, topic_num in enumerate(top_index):    topic_dict[topic_num]=topic_names[i] df_pred = topic_attribution(tokenized_data, mgp, topic_dict, threshold=0.4) 
One might ask what is the threshold input parameter of the topic_attribution function. Actually, the topics are allocated to a text given a probability and topic_attribution is a custom function that allows to choose which threshold (confidence degree) to consider in order to belong to a topic. For example, looking at the highest probability allocation of a topic to a text, if this probability is below 0.4 the text will be allocated in a “Others” topic.
NB: This custom topic_attribution function is built upon the original function available in the GSDMM package: choose_best_label, which outputs the topic with the highest probability to belong to a document.
Now it’s time to allocate the topic found to the documents and compare them with the ground truth (✅ vs ❌)
df_pred[['content', 'topic_name', 'topic_true_name']].head(20)
Naively comparing the predicted topics to the true topics we would have had a 82% accuracy! 🎯Only with a 9 words average by document, a small corpus of 1705 documents and very few hyper-parameters tuning!
It’s great to have an efficient model but it is even better if we are able to simply show and interact with its results. To do so, pyLDAvis is a very powerful tool for topic modeling visualization, allowing to dynamically display the clusters and their content in a 2-D space dimension.
Now it’s your turn to try it on your own data (social media comments, online chats’ answers...) 💪
Et voilà ! 👌
I would like to thank Rajaa El Hamdani for reviewing and giving me her feedback.
References and other useful resources- The original paper of GSDMM - A nice python package that implements STTM.- The pyLDAvis library to beautifully visualize topics in a bunch of texts (or any bag-of- words alike data).- A recent comparative survey of STTM to see other strategies.
PS: For those willing to dive deeper in STTM, there is an interesting further approach (which I have not personally explore for now) called GPU-DMM that has shown SOTA results on Short Text Topic Modeling tasks. In short, GPU-DMM is using pre-trained word embeddings as an external source of knowledge to influence the sampling of words to generate topics and documents.