NLP Preprocessing and Latent Dirichlet Allocation (LDA) Topic Modeling with Gensim | by Sejal Dua | Towards Data Science

In my last article, I walked through how to extract metadata from email newsletters. Specifically, I demonstrated how to use the Gmail API to collect Title, Subtitle, Author, Publication, and Minutes (an estimate of article length in reading time) for each suggested article in my Medium Daily Digest newsletter. With this data, my goal is to investigate: 1) how Medium’s recommendation engine seems to know my reading interests so well and 2) how my interests have evolved over time.
Topic modeling is an unsupervised NLP technique used to identify recurring patterns of words from a collection of documents forming a text corpus. It can be useful for discovering patterns across a collection of documents, organizing large blocks of textual data, information retrieval from unstructured text, and more. In its most powerful form, it can be used to boost recommendation engines with more signal pertaining to user behavior, and it can also be used gain insight into the inner workings of a recommendation systems, which is what we will attempt to accomplish through this article.
So how does it work? By detecting patterns such as word frequency and distance between words, a topic model clusters documents that are similar, according to words and expressions that appear most often. With this information, you can quickly deduce which texts are associated with your topics of interest.
Topics can be defined as “a repeating pattern of co-occurring terms in a corpus”. A good topic model might infer that the words “health”, “doctor”, “patient”, “hospital” all fall under the umbrella of “Healthcare”. Words like “neural network”, “back propogation”, “epoch”, and “loss” might be designated into the “Deep Learning” topic bin. However, it is important to note that since topic modeling is an unsupervised approach, the model can learn that certain words are associated with one another, but it has not been trained on labeled data to learn that “health”, “doctor”, “patient”, “hospital” are associated with “Healthcare” as a known category. Instead, it is up to the user to either gain insights from the unlabeled groupings of similar documents — or to train a topic classification model on text containing categorical topic labels if those are needed as output.
There are many technical methodologies that can be used to perform topic modeling. Some of these include Latent Dirichlet Allocation (LDA), TextRank, Latent Semantic Analysis (LSA), Non-negative Matrix Factorization (NMF), Pachinko Allocation Model (PAM), and others. In this article, we will focus on implementing Latent Dirichlet Allocation, which is the most common method.
LDA is a matrix factorization technique that was developed using the Variational Exception Maximization (VEM) algorithm. LDA is built atop the premise that each document can be described by the probabilistic distribution of topics and each topic can be described by the probabilistic distribution of words. This can give us a much clearer vision of how topics are connected.
In vector space, any corpus or collection of documents can be represented as a document-word matrix consisting of N documents by M words. The value of each cell in this matrix denotes the frequency of word W_j in document D_i. The LDA algorithm trains a topic model by converting this document-word matrix into two lower dimensional matrices, M1 and M2, which represent document-topic and topic-word matrices, respectively. If you take anything away about how LDA works on a mathematical level — and seriously don’t stress about this because unless you are a researcher or into computer science theory, we can just import a handy Python package, create an LDA object, and call it a day — try to understand that LDA relies on Bayes’ theory and two key probability computations to update the probability for a given word w belonging to topic t:
p(topic t| document d) — the proportion of words in document d that are assigned to topic t
p(word w | topic t) — captures how many documents are in topic t because of word w
From Part 1 of this series, we were able to obtain the dataset below. After removing duplicates, we are left with 7.1k rows. Check out the Datapane embed below!
Before we can begin with any topic modeling, let’s make sure we install and import all the libraries we will need.
# Essentialsimport base64import refrom tqdm import tqdmimport numpy as npimport pandas as pdimport matplotlib.pyplot as pltimport datapane as dpdp.login(token='INSERT_TOKEN_HERE')# Gensim and LDAimport gensimimport gensim.corpora as corporafrom gensim.utils import simple_preprocessfrom gensim.models import CoherenceModelfrom gensim.parsing.preprocessing import STOPWORDSimport pyLDAvisimport pyLDAvis.gensim  # don't skip this# NLP stuffimport contractionsimport demojiimport stringimport nltkfrom nltk.stem import WordNetLemmatizer, SnowballStemmerfrom nltk.stem.porter import *from nltk.corpus import stopwordsstop_words = stopwords.words('english')nltk.download('wordnet')import spacy# Plotting toolsfrom bokeh.plotting import figure, output_file, showfrom bokeh.models import Labelfrom bokeh.io import output_notebookimport matplotlib.colors as mcolorsimport matplotlib.pyplot as plt%matplotlib inline# Miscellaneousfrom sklearn.manifold import TSNEfrom pprint import pprint
One of the most critical steps in any NLP-related project is column-wise preprocessing of all documents. While there are some standard cleaning methods that, in general, can be applied to any corpus, there are also a considerable amount of intuition-driven decisions that we must make. For instance, does it make sense to emojis? What about stop words? Should numbers be left as is, converted to text, or removed entirely? Should words that occur in more than 80% of all documents be ignored or investigated? Should variations of the same root word be stemmed or lemmatized?
The function above takes a pandas series object as input and applies lambda functions which convert the text to lowercase, remove emojis, expand contractions, remove punctuation, remove stop words, perform lemmatization, and remove words that are too short (less than 3 characters long).
When it comes to stopword removal, many presume that this is a pragmatic preprocessing step that will always do more good than harm. However, I highly urge you to actually print out whatever list of stopwords you decide to use (NOTE: I use the nltk.corpus.stopwords.words(‘english’) set) and make context-aware tweaks to optimize your corpus. A common example of a stopword that requires a bit of deliberation before removing is “not”. When a textual data source consists of opinion-based commentary from users or other detailed descriptions, removing “not” tends to obfuscate a major source of signal in the data. Here, we are just looking at titles and subtitles of medium articles, so it does not matter so much whether we decide to keep “not” or discard it, but I will keep it on principle.
For those unfamiliar with lemmatization and stemming, you can think of lemmatization as the process of grouping together words with the same root or lemma but with different inflections or derivatives of meaning — words in third person are changed to first person and verbs in past and future tenses are changed into present. Stemming is the process of reducing inflection in words to their root forms by removing suffixes or prefixes. Unlike stemming, lemmatization ensures that the root word or lemma is a valid word belonging to the English language. For example, runs, running, and ran all would map to the lemma run. To illustrate the difference between lemmatization and stemming, the stem of the words intelligence and intelligent is intellig, which is not actually a valid English word.
An n-gram is a contiguous sequence of n items from a given sample of text or document. Bigrams are two words frequently occurring together in the document (e.g. data_science, machine_learning, sentiment_analysis). Trigrams are 3 words frequently occurring (e.g. towards_data_science, natural_language_processing, exploratory_data_analysis)
Gensim’s Phrases model can build and implement bigrams, trigrams, quadgrams, and more. The two important arguments to Phrases are min_count and threshold. The higher the values of these parameters, the harder it is for words to be combined as bigrams and trigrams.
The gensim Python library makes it ridiculously simple to create an LDA topic model. The only bit of prep work we have to do is create a dictionary and corpus.
A dictionary is a mapping of word ids to words. To create our dictionary, we can create a built in gensim.corpora.Dictionary object. From there, the filter_extremes() method is essential in order to ensure that we get a desirable frequency and representation of tokens in our dictionary.
id2word = corpora.Dictionary(data_preprocessed)id2word.filter_extremes(no_below=15, no_above=0.4, keep_n=80000)
The filter_extremes() method takes 3 parameters. Let’s break down what those mean:
filter out tokens that appear in less than 15 documents
filter out tokens that appear in more than 40% of documents
after the above two steps, keep only the first 80,000 most frequent tokens
A corpus is essentially a mapping of word ids to word frequencies. For each document, a bag of words representation is created:
corpus = [id2word.doc2bow(text) for text in texts]
Using the dictionary and corpus, we can obtain a human-readable word-frequency mapping of one sample document:
[[('better', 1),  ('creating', 1),  ('data_visualization', 1),  ('design', 1),  ('principle', 1),  ('towards', 1)]]
Pretty neat... also notice how one of our bigrams (data_visualization) just made an appearance!
The code to build our LDA model is, believe it or not, a one-liner.
lda_model = gensim.models.ldamodel.LdaModel(corpus=corpus,                                           id2word=id2word,                                           num_topics=10,                                            random_state=100,                                           update_every=1,                                           chunksize=100,                                           passes=10,                                           alpha='auto',                                           per_word_topics=True)
We already covered the corpus and dictionary parameters. We will fix num_topics to be 10 initially. random_state is pretty self-explanatory and does not have a huge impact on the model; it is simply used to obtain results that do not change with each execution of the notebook. update_every determines how often the model parameters should be updated. chunksize is the number of documents to be used in each training chunk. passes is the total number of training passes. The last parameter, per_word_topics, when set to True, computes a list of topics, sorted in descending order of most likely topics for each word, along with their phi values multiplied by the feature length (i.e. word count).
Now that we have our “trained” topic model (remember: this is an unsupervised approach... think of it more like clustering than anything else), let’s print the top 10 keywords in our 10 topics.
Interesting, so python is obviously a prevalent keyword in this corpus, and rightfully so! It looks like we have a topic about learning good design, one about machine learning articles involving twitter data (perhaps?), how to be successful in research, productivity tips, developer stories, etc. This is a good start, but it is not providing us with a satisfying, holistic picture just yet.
The next question you may be asking is how do we know how good our model actually is, just from that one line of code to build it?
There are two evaluation metrics that are typically used in the domain of topic modeling: perplexity and coherence.
Shashank Kapadia explains these metrics better than I ever could in his article “Evaluate Topic Models: Latent Dirichlet Allocation (LDA)”. He defines them as the following:
Perplexity captures how surprised a model is of new data it has not seen before, and is measured as the normalized log-likelihood of a held-out test set.
Coherence measures the degree of semantic similarity between high scoring words in the topic. If you are interested, the c_v measure is based on a sliding window, one-set segmentation of the top words and an indirect confirmation measure that uses normalized pointwise mutual information (NPMI) and cosine similarity.
To compute these values, just execute the following code:
The initial perplexity and coherence of our vanilla LDA model are -6.68 and 0.4, respectively. Going forward, we will want to minimize perplexity and maximize coherence.
Now you might be wondering how we can visualize our topics aside from just printing out keywords or, god forbid, another wordcloud. Well, fortunately we have been given the gift of pyLDAvis! pyLDAvis is a Python library for interactive topic model visualization. In just one small block of code, we can obtain these interactive subplots depicting the distance between topics on a 2D plane and the top 30 most relevant and salient terms within the topic (both of these metrics are defined mathematically in the bottom right corner of the visualization below).
From the GIF above, it is apparent that topics 1 and 2 are the most separated from each other, and from the remaining 8 topics, which are all generally similar to one another. While this is a super nifty visualization method to help make sense of which topics are most closely related and which words carry the strongest amount of signal, the current model does not appear to be very smart. In other words, hovering over each topic does not entirely help us make sense of all of the topics represented in this corpus. This is an indicator that it may be time to go back to the drawing board and try out a more sophisticated LDA model.
MALLET, written by Andrew McCallum, is a Java-based package for statistical natural language processing, document classification, clustering, topic modeling, information extraction, and other machine learning applications to text. The MALLET topic modeling toolkit contains efficient, sampling-based implementations of Latent Dirichlet Allocation. The main optimization difference is that Gensim’s (vanilla) LDA uses a Variational Bayes sampling method which is faster but less precise that Mallet’s Gibbs Sampling.
Before we start using it with Gensim for LDA, we must download the mallet-2.0.8.zip package on our system and unzip it. The package to download and more detailed instructions can be found here. Once installed and unzipped, set the mallet_path variable to the file path where your download currently resides on your local machine. Once this is done, we are ready to build our LDA Mallet model!
Python provides Gensim wrapper for Latent Dirichlet Allocation (LDA). The syntax of that wrapper is gensim.models.wrappers.LdaMallet. This module, allows LDA model estimation from a training corpus and inference of topic distribution on new, unseen documents as well.
mallet_path = '~/Downloads/mallet-2.0.8/bin/mallet' # update this pathldamallet = gensim.models.wrappers.LdaMallet(mallet_path,                                              corpus=corpus,                                              num_topics=10,                                             id2word=id2word,                                             random_seed=42,                                             workers=6)
Initially, we can set the number of topics to be 10 in order to obtain an apples-to-apples comparison of Gensim’s LDA versus LDA Mallet. It is not possible to obtain a perplexity score from our LdaMallet model, but it is possible to get a coherence score. Below is some code to preview the topics and compute the coherence score:
Next, let’s programmatically determine the most optimal number of topics into which we will partition our documents. In order to discern this number, we will want to test a few models with increasing numbers of topics and then look for the least number of topics that yields a high enough coherence value.
From the graph above, we can see our coherence score shoot up to around 0.38, then to around 0.43, and then to around 0.45. It is up to the user to pick a number of topics that they deem suitable. I will go ahead and select the second “elbow point” at x = 24 topics, which corresponds to a coherence score of about 0.43.
We can preview our topics as we did before, but we can also cut right to the chase and make a pyLDAvis. In order to accomplish this, we actually have one additional step to perform: creating a regular LDA model with the learned parameters of our Mallet model so that we can pass the regular gensim LDA object into pyLDAvis. This can be done with a simple function which is included in the full notebook if you would like to follow along.
The intertopic distance map above is a bit more interesting than the one for our initial model! We can see more topics scattered about the two-dimensional plane. In particular, there are 2–3 clusters of topics worth analyzing. Disregarding the randomly assigned topic labels in the visualization, the 3 topics in the lower left quadrant appear to consist primarily of Medium articles that fall under the umbrella of Python tutorials, guides, and introductions. The top rightmost two topics are characterized by their less technical keywords and seem to pertain more toward business, marketing, and design trends and skillsets. The bottom rightmost topics seem to pertain towards building visualizations and data-centric web apps.
From here, we can find the most dominant topic for each article title and subtitle, and we can also get the degree to which the document is represented by the dominant topic. Those new attributes are included in the data table below:
We can also gain a better understanding of each topic by getting its number of documents, keywords, and most representative document.
The code to wrangle a DataFrame of topic information with these key attributes is provided above. Note that it is important to continually reset the index to ensure that the columns are aligned in accordance with the appropriate topic number. The merge method for pandas DataFrame can be leveraged here.
From the above table containing features that describe each topic, we can learn that topic 0 contains the most articles. The topic containing the second most amount of documents is unfortunate but also quite intuitive, articles which talk about the “pandemic”, “covid”, and/or “coronavirus.” The “Representative_Doc” column should be taken with a grain of salt, but it is useful to get a sense of how a title string might include a subset of relevant topic keywords given some out-of-topic domain context.
At this point in the project, we have arrived at a sufficiently functional topic model that leverages strong sources of signal to cluster or separate our documents in a sensical way, which we have validated by inspecting keywords, inter topic distance, and representative documents. It can become very tempting to resort to topic-specific wordclouds as a means of summarizing each topic further, but I would like to make the case that a better visualization technique is needed.
... and if, for some reason, you still lean toward wordclouds, perhaps we can at least reach a consensus that as the number of topics increases, visually inspecting wordclouds gets more and more overwhelming. So if you are ready to graduate from wordclouds, take my hand and come with me. Let’s find a better visualization technique.
t-Distributed Stochastic Neighbor Embedding (t-SNE), developed by Laurens van der Maaten, is a non-linear technique for dimensionality reduction that is particularly well suited for the visualization of high-dimensional datasets. It is extensively applied in image processing, NLP, genomic data and speech processing.
So how does it work and why should we use it? In simple terms and according to a really helpful DataCamp introduction, t-SNE minimizes the divergence between two distributions: a distribution that measures pairwise similarities of the input objects and a distribution that measures pairwise similarities of the corresponding low-dimensional points in the embedding. In this way, t-SNE maps the multi-dimensional data to a lower dimensional space and attempts to find patterns in the data by identifying observed clusters based on similarity of data points with multiple features.
Using our 24-topic Mallet model, we can see how the topics are pretty well clustered on the two-dimensonal plane. Furthermore, the size of each circle representing a single article is encoded by the topic representation percentage. Thus, larger circles are more strongly associated with the topic to which they belong — hotspots or clusters of these larger circles can be easily identified as related articles which likely share a cohesive theme. We can also hover over each circle to see the article title, subtitle, and topic representation percentage.
One drawback of this approach is that after this process is performed, the input features are no longer identifiable. Hence it is mainly a data exploration and visualization technique. Moreover, it is difficult to obtain an at-a-glance understanding of our topics from this t-SNE scatterplot without either spending a significant amount of time hovering over the circles or having an internalized understanding of topic number to keyword mappings.
While we certainly made progress in understanding which general topics my suggested Medium articles cover, my initial research question sought to explore how my reading interests have evolved with respect to time. I will conclude this article by stating that my quest to implement a topic model which yields temporal insights is still incomplete! In my next article, I will introduce BERTopic and share with you all how my reading interests and reading behavior have gone through some interesting shifts over the last year or two.
Until then, happy reading, learning, and exploring! I hope you learned a thing or two about topic modeling with Gensim through this comprehensive tutorial. All the code can be found in the GitHub repository linked below.