Don’t be Afraid of Nonparametric Topic Models (Part 2: Python) | by Eduardo Coronado Sroka | Towards Data Science

This article builds upon high-level foundational material I covered in my previous article and describes how to implement a Hierarchical Dirichlet Process model for topic modeling in Python.
Let us all agree, it’s one thing to learn and talk about cool, new methods and another to actually implement/test them with data. Mostly because learning about them doesn’t come with the typical frustrations that arise with bugs, weird errors, etc. However, I personally believe that what comes out of all that tinkering is a deeper understanding of the concept itself.
And guess what? Bayesian nonparametric (BNP) methods such as Hierarchical Dirichlet Processes (HDP) aren’t the exception.
Before you think I’m about to throw you in at the deep end of the coding pool, don’t fret. I wrote this article with the overall goal that by the end you can confidently implement an HDP model that drives value in your projects (or allows you to humble brag to your friends).
Here’s what I’ll cover
A step-by-step tutorial on how to implement an HDP model using an existing Python library
How its performance compares to Latent Dirichlet Allocation (LDA) models
Some key considerations when implementing an HDP, pitfalls and potential fixes
I’ll be using the 20 Newsgroup dataset available through sklearn.datasets. This is a great toy dataset given the collection of approximately 20,000 documents is split up almost evenly across 20 different topics (newsgroups). Thus, in some sense we already know the true topics the models should infer.
The article is broken down into the following sections
Data PreprocessingHDP Model Training and EvaluationModel ComparisonCautionary Tales
Data Preprocessing
HDP Model Training and Evaluation
Model Comparison
Cautionary Tales
Before we dive in I’d recommend you install the following dependencies spaCy , nltk , gensim , tomotopy , plotnine , and wordcloud. You can either install each individually using pip or you can use the requirements file I created to make your life easier as follows
pip3 install -r requirements.txt
If you haven’t preprocessed text data for NLP projects, I highly recommend you check out this step-by-step tutorial beforehand
Let’s start by first loading the 20 Newsgroup data set, specifying that we only want the train subset for our model.
from sklearn.datasets import fetch_20newsgroups# Read in train subset (11,314 observations)news = fetch_20newsgroups(subset='train')
This news instance contains both text data (news.data) and labels (news.target and news.target_names ) which in pandas it might look like
Rather than boring you with a step-by-step on how I preprocessed the content data with standard methods (e.g. tokenization, stop word removal, etc.), I will instead summarize these steps below.
As we move forward in this section (and in future projects) I’d like to ask that you keep the following in mind:
Do these preprocessing steps make sense based on the end goal(s) and modeling method of choice?
This can easily be overlooked given standard preprocessing methods work quite well in many applications. For example, in our case we want to remove noise generated by stop words (on, and, etc.) so that our models can better capture latent topics that resemble the true topics. However, is this always the case? No.[Check out this article on why this isn’t a good idea in sentiment analysis]
First, I removed any special characters inherent to this data set such as @ and \n , as well as single quotes ' using regular expression substitutions.
Here I used the function gensim.simple_preprocess which very efficiently tokenizes each document (i.e. splits the text into individual words). To remove any potential accents, I ran it with the deacc=True parameter.
These are helpful to build given they account for word co-occurrences. For example, in our data this would generate ['oil_leak'] instead of ['oil', leak'] . You can use gensim ‘s built-in Phrases and Phraser functions to achieve this.
Inferring latent topics will be easier if we first prune the documents of non-informative words (i.e. stop words). I downloaded nltk ‘s English stop words and added few simple ones unique to this data set ['from', ‘subject', ‘re','edu',use'] .
Given the data set is relatively small (i.e. ~10k observations) I implemented a lemmatization scheme for nouns, verbs, adjective and adverbs using spaCy and part-of-speech (POS) tags. Simply put, I removed any inflectional endings and returned the base/dictionary word as shown below.
"A letter has been written asking him to be released"[ex. Original ==> Lemmatized, POS tag]A        ==> a, DETletter   ==> letter, NOUNhas      ==> have, AUXbeen     ==> be, AUXwritten  ==> write, VERBasking   ==> ask, VERBhim      ==> -PRON-, PRONto       ==> to, PARTbe       ==> be, AUXreleased ==> release, VERB
Although lemmatization provides better tokens, it does at the expense of potentially taking longer than you’d like to sit waiting for it to finish. Thus, an alternative when time is of essence or if you have a large data set is stemming which uses crude heuristics to chop off the ends of words in hopes of getting the base word. (For example, in this data set lemmatization took about 4 mins while stemming took only 14 secs).
I built a custom script newsgrp_preprocess(link here) that incorporates all the above steps and outputs ready-to-use data for our HDP model with over 1M tokens (i.e.word_list_lemmatized). Additionally, it tidy-ups the information in the news instance and outputs the information in a nice pandas dataframe (shown earlier in the article).
> from scripts.newsgrp_preprocess import run_preprocess> news_df, word_list_lemmatized = run_preprocess(news)# Showing first document, first seven tokens> word_list_lemmatized[0][:7] > ['where', 'thing', 'car', 'nntp_poste', 'host', 'park', 'line']
Now that that’s done, let’s move on to the really cool stuff — training the model!
If you’re not familiar with Bayesian models, you might ask “What does it mean to train a Bayesian model”? Well, it basically means we’re trying to infer/learn a distribution. In our case, we’re trying to learn the distribution of unobserved (latent) topics from the documents.
To train a Bayesian model you commonly use methods that fall within two main camps: Monte Carlo methods (e.g. Gibbs/MCMC sampling) and Approximation/Optimization methods (e.g. Variational Inference).
MCMC? Variational...what? Don’t worry, here’s an easy read by Joseph Rocca that explains these methods even if you don’t know much Bayesian stats
Without going into much detail, both camps achieve basically the same goal: a set of inferred topics from the data. However, each has advantages and disadvantages (some of which I’ll cover in later sections) that are important when considering which method to use.
For the rest of this section, I’ll be using Python libraries that use a Monte Carlo method called Collapsed Gibbs sampling. Compared to a traditional Gibbs sampler, this method speeds up the topic inference process (i.e. model training).
Here I used the tomotopy Python library.
If you’ve had any experience implementing HDPs, at this point you might ask, “But why didn’t you use thegensim.HdpModelfunction?” Well, I tried (trust me, real hard). Remember the frustrations I mentioned at the beginning? Well for me this was one of them because even after extensive tuning I wasn’t able to get gensim’s method to generate quality results with this dataset.
Training a tomotopy model is quite simple. First you initiate a model object by setting some parameters like how the model will weight tokens, thresholds related to token frequency, and the HDP model’s concentration parameters alpha and gamma (see left).
For this dataset, I restricted the model to using only words that appeared in at least 5 documents with the min_cf argument while excluding the 7 most frequent words withrm_top. Similarly, I set the concentration parameters gamma=1 and alpha=0.1 given I assume documents share many topics while individual documents only talk about few topics. I initialized the number of topics withinitial_k=10 which acts as a sort of prior. I chose this given the data’s 20 topics are grouped in 6 overarching groups (e.g. recommendation under rec.car , rec.bike ) and I assumed the misc group could have some additional topics that should be accounted for.
import tomotopy as tpterm_weight = tp.TermWeight.ONEhdp = tp.HDPModel(tw=term_weight, min_cf=5, rm_top=7, gamma=1,                  alpha=0.1, initial_k=10, seed=99999)
Once we have the hdp object instantiated, we can add the documents it will use to train the model as follows. (In case it’s helpful, I’ve automated these steps with a custom function train_HDPModel.)
The above example output provides useful early-diagnostic information. For example, seeing the per-word log-likelihood increasing tells us the model is learning adequately.
Extracting topics from the model isn’t as straightforward as other packages, so I’ve built a custom script get_hdp_topics to ease this process.
The following section will sound tedious, however I highly encourage you to walk through it. It covers best practices on how to evaluate these kind of models.
Given topic models are unsupervised methods we are unable to use common performance metrics (e.g. RMSE) to evaluate them, instead we use a metric called coherence which provides an objective measure of whether words grouped together as a topic make sense.
There are multiple ways to compute this metric but basically, a topic is said to have high coherence if the words defining a topic have a high probability of appearing together (co-occur) across documents. In general, the CV method is preferred given it takes into account how close words appear together through a sliding window to compute these probabilities.
Given we know the topics in our data set, we can evaluate two things:
Whether the model’s topics represent the true topics (coherence)How well does the model infer the topics of an unseen (out-of-sample) documents
Whether the model’s topics represent the true topics (coherence)
How well does the model infer the topics of an unseen (out-of-sample) documents
Given tomotopy offers three distinct token weighting schemes, I tested each to compare their performance based on the above criteria. First, I compared their coherence using gensim.CoherenceModel with coherence='c_v' using a custom script. The CV metric scores range from 0 to 1 (where good topic coherence scores range between 0.5–0.65) .
For this dataset the Inverse Document Frequency model seems to perform the best based on topic coherence. If your goal is to only understand some latent topics from a specific dataset, then great — you’re done! Choose the model with highest coherence. However, be aware this does not necessarily mean the model will generalize well (i.e. accurately assign topics to unseen documents).
In our case, we have a labeled test set we can use to verify how the models generalize. (If you don’t have labeled data you can do something similar as a sanity check. Grab an unseen document to predict on, select the most dominant topic assigned and see if that assignment makes sense based on the text.)
gets predicted to one You can also use the training set as a sanity check if you’d like even if it’s unlabeled data, just predict on a document and see if it makes sense based on the content.)
The word clouds below provide a great example. Using the same steps as before, we get a test set (subset='test') and evaluate how well each model generalizes. We see that the model with the highest coherence (IDF) doesn’t necessarily assign the “correct” topic (rec.autos), it instead seems to think this document talks about computers.
Although this was a very specific example, I chose to continue with the HDP IDF model given it tended to produce topics that resembled the true labels and assigned the right topics to test documents more often than the others.
As a quick side note, topic labels can be subjective (i.e. one person might interpret the words as referring to a hardware topic while another to a software topic). To avoid that tomotopy offers an interesting method that objectively labels the topics. You can find some examples here.
Congratulations! If you’ve made it this far you pretty much know how to implement an HDP model in Python! However, how does it compare to LDA?
Let’s compare our HDP model versus a MALLET LDA model (fun!). I used this version instead of the default gensim LDA because it allows an apples-to-apples comparison (i.e. it also uses a Collapsed Gibbs sampler). In this case gensim does have an easy wrapper LdaMallet to quickly implement this model once you’ve downloaded the MALLET binary. To understand how to run it I suggest you look at the following Jupyter notebook .
Using the same dataset, I compared several LDA models’ performance as above (i.e. coherence + generalizability) to those of the HDP models. Below (left), we can see that topic coherence increases as we increase the topic parameter in the LDA model.
This comparison demonstrates how without specifying topics off-hand HDP models can achieve similar or higher topic coherence than LDA models
Similarly, we can see that our best HDP model (IDF) has similar performance to the best LDA model (topics=26) in terms of assigning the correct topic to an unseen documents (right).
What is a Data Science article without some good ol’ cautionary tales. Let’s face it every modeling technique has advantages and disadvantages, and there are situations were things can go wrong. Here I just want to share a couple of things to keep in mind when implementing HDP models (or topic models in general).
Really think about your preprocessing steps. An easy way to start is to consider what problem you’re aiming to solve is, the data you have, and the model you’ll use. Many times we can just go with the flow and use similar preprocessing steps across projects, however as I mentioned above this might lead to bad outputs.
For example, the goal of this project was to implement a model that could learn 20(ish) topics from text data. Here it made sense to remove stop words to reduce noise and lemmatize given the data set size. However, if the goal were to do some sentiment analysis then removing stop words wouldn’t have been a good choice.
Coherence, as one of many metrics you can build to evaluate topic models, provides an intuitive way to gauge performance. I don’t know about you, but I think sometimes we tend to fixate on statistics (e.g. RMSE or classification for predictive models), especially when they seem to indicate good performance.
Here’s a clear example. Remember that we took coherence as a good indicator of performance? Well, this was certainly not true when I used gensim.HdpModel. Since their implementation is based on variational inference, I tested (as one of hundreds of tuning combinations) how changing one of the learning parameters (kappa) would affected the model’s topic coherence and performance on unseen documents.
The result was that as I increased kappa the model’s coherence score increased. You might ask, “Well great, that’s what we wanted right?” Yes, but in this case, when you try to test the model on an unseen document you observe that 1) every single document is assigned a the same topic and 2) this topic is so general that you wouldn’t be able to make sense of it. The model had converged to a solution that performed “well” (e.g. a local optima) by the given metric but which nonetheless wasn’t useful.
Therefore, if there is one key thing I’d like for you to learn from this article is
Before fully trusting a statistic, take a step back to think whether that number makes sense and why it makes sense
Previously I mentioned there were two main camps used in Bayesian inference: Monte Carlo methods and variational methods. Choosing which package implementation to use varies greatly on your specific project and goals. However, I believe it boils down to a speed/memory vs. accuracy trade-off.
tomotopy models: Fast and accurateAs previously mentioned this packages uses a Collapsed Gibbs sampler. The main advantage is that it inherently produces unbiased and accurate results. However, this method doesn’t scale well (memory costs increase linearly with the number of observations). In our case it made sense but if you were to take a dataset of 1M documents you will probably be waiting a while for it to finish running.
gensim models: Faster and scalableThis package uses an online variational inference method which, to keep it simple, allows you to use classic variational inference approximation tools at scale. The main advantages of this method are speed and low memory consumption (memory doesn’t grow linearly with the number of observations, so this is great for large datasets). However, it gains speed at the expense of accuracy given you are approximating a target distribution with usually a “simpler distribution” rather than sampling directly from the target as in Collapsed Gibbs sampling. This not only introduces bias but also might require extensive parameter tuning to get useful results. Below is an example of how two parameters affect the learning rate and thus convergence (you can access the Dash app here).
(Given these limitations new methods have been developed to try to address the approximation bias of variational inference.)
HDP models are powerful alternatives to LDA models when you don’t wish to specify topics beforehand and there are several packages out there that can help you easily implement them. However, remember that some due diligence before, during, and after implementing these models can ensure your results are in line with your initial objectives.
So, now that you’ve learned how to implement an HDP model (and count fluently in Spanish), go ahead and test your newly learned skills!
Wang, Chong, John Paisley, and David M. Blei. “Online variational inference for the hierarchical Dirichlet process.” Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics, in PMLR. 2011.Griffiths, Thomas L., and Mark Steyvers. “Finding scientific topics.” Proceedings of the National academy of Sciences 101.suppl 1 (2004): 5228–5235.Tomotopy. https://github.com/bab2min/tomotopy . https://doi.org/10.5281/zenodo.3816629Bryant, Michael, and Erik B. Sudderth. “Truly nonparametric online variational inference for hierarchical Dirichlet processes.” Advances in Neural Information Processing Systems. 2012.
Wang, Chong, John Paisley, and David M. Blei. “Online variational inference for the hierarchical Dirichlet process.” Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics, in PMLR. 2011.
Griffiths, Thomas L., and Mark Steyvers. “Finding scientific topics.” Proceedings of the National academy of Sciences 101.suppl 1 (2004): 5228–5235.
Tomotopy. https://github.com/bab2min/tomotopy . https://doi.org/10.5281/zenodo.3816629
Bryant, Michael, and Erik B. Sudderth. “Truly nonparametric online variational inference for hierarchical Dirichlet processes.” Advances in Neural Information Processing Systems. 2012.
If you liked the article feel free to share! Comment or tweet (@ecoronado92) if you have any questions or you see anything incorrect/questionable.
All code and notebooks used in this article can be found here