Interactive Topic Modeling with BERTopic | Towards Data Science

Every day, businesses deal with large volumes of unstructured text. From customer interactions in emails to online feedback and reviews. To deal with this large amount of text, we look towards topic modeling. A technique to automatically extract meaning from documents by identifying recurrent topics.
A few months ago, I wrote an article on leveraging BERT for topic modeling. It blew up unexpectedly and I was surprised by the positive feedback I had gotten!
towardsdatascience.com
I decided to focus on further developing the topic modeling technique the article was based on, namely BERTopic.
BERTopic is a topic modeling technique that leverages BERT embeddings and a class-based TF-IDF to create dense clusters allowing for easily interpretable topics whilst keeping important words in the topic descriptions.
I am now at a point where BERTopic has gotten enough traction and development that I feel confident it can replace or complement other topic modeling techniques, such as LDA.
github.com
The main purpose of this article is to give you an in-depth overview of BERTopic’s features and tutorials on how to best apply this for your own projects.
As always, we start by installing the package via pypi:
pip install bertopic
To use the visualization options, install BERTopic as follows:
pip install bertopic[visualization]
Using BERTopic out-of-the-box is quite straightforward. You load in your documents as a list of strings and simply pass it to the fit_transform method.
To give you an example, below we will be using the 20 newsgroups dataset:
There are two outputs generated, topics and probabilities. A value in topics simply represents the topic it is assigned to. Probabilities on the other hand demonstrate the likelihood of a document falling into any of the possible topics.
Next, we can access the topics that were generated by their relative frequency:
In the output above, it seems that Topic -1 is the largest. -1 refers to all outliers which do not have a topic assigned. Forcing documents in a topic could lead to poor performance. Thus, we ignore Topic -1.
Instead, let us take a look at the second most frequent topic that was generated, namely Topic 49:
Since I created this model, I am obviously biased, but to me, this does seem like a coherent and easily interpretable topic!
Under the hood, BERTopic is using sentence-transformers to create embeddings for the documents you pass it. As a default, BERTopic is set to using an English model but is supports any language for which an embedding model exists.
You can choose the language by simply setting the language parameter in BERTopic:
When you select a language, the corresponding sentence-transformers model will be loaded. This is typically a multilingual model that supports many languages.
Having said that, if you have a mixture of language in your documents you can use BERTopic(language="multilingual") to select a model that supports over 50 languages!
To chose a different pre-trained embedding model, we simply pass it through BERTopic by pointing the variable embedding_model towards the corresponding sentence-transformers model:
Click here for a list of supported sentence transformers models.
We can easily save a trained BERTopic model by calling save:
Then, we can load the model in one line:
Now that we have covered the basics and generated our topics, we visualize them! Having an overall picture of the topics allows us to generate an internal perception of the topic model’s quality.
To visualize our topics I took inspiration from LDAvis which is a framework for visualizing LDA topic models. It allows you to interactively explore topics and the words that describe them.
To achieve a similar effect in BERTopic, I embedded our class-based TF-IDF representation of the topics in 2D using Umap. Then, it was a simple matter of visualizing the dimensions using Plotly to create an interactive view.
To do this, simply call model.visualize_topics() in order to visualize our topics:
An interactive Plotly figure will be generated which can be used as indicated in the animation above. Each circle indicates a topic and its size is the frequency of the topic across all documents.
To try it out yourself, take a look at the documentation here where you can find an interactive version!
For each document, we can also visualize the probability of that document belong to each possible topic. To do so, we use the variable probabilities after running BERTopic to understand how confident the model is for that instance.
Since there are too many topics to visualize, we visualize the probability distribution of the most probable topics:
It seems that for this document, the model had some more difficulty choosing the correct topic as they were multiple topics very similar to each other.
As we have seen before, hundreds of topics could be generated. At times, this might simply be too much for you to explore or a too fine-grained solution.
Fortunately, we can reduce the number of topics by merging pairs of topics that are most similar to each other, as indicated by the cosine similarity between c-TF-IDF vectors.
Below, I will go into three methods for reducing the number of topics that result from BERTopic.
When initiating your BERTopic model, you might already have a feeling of the number of topics that could reside in your documents.
By setting the nr_topics variable, BERTopic will find the most similar pairs of topics and merge them, starting from the least frequent topic, until we reach the value of nr_topics:
It is advised, however, to keep a decently high value, such as 50 to prevent topics from being merged that should not.
As indicated above, if you merge topics to a low nr_topics topics will be forced to merge even though they might not actually be that similar to each other.
Instead, we can reduce the number of topics iteratively as long as a pair of topics is found that exceeds a minimum similarity of 0.9.
To use this option, we simply set nr_topics to "auto" before training our model:
What if you are left with too many topics after training which took many hours? It would be a shame to have to re-train your model just to experiment with the number of topics.
Fortunately, we can reduce the number of topics after having trained a BERTopic model. Another advantage of doing so is that you can decide the number of topics after knowing how many are actually created:
Using the code above, we reduce the number of topics to 30 after having trained the model. This allows you to play around with the number of topics that suit your use-case!
Topics are typically represented by a set of words. In BERTopic, these words are extracted from the documents using a class-based TF-IDF.
At times, you might not be happy with the representation of the topics that were created. This is possible when you selected to have only 1-gram words as representation. Perhaps you want to try out a different n-gram range or you have a custom vectorizer that you want to use.
To update the topic representation after training, we can use the function update_topics to update the topic representation with new parameters for c-TF-IDF:
We can also use a custom CountVectorizer instead:
Why limit ourselves to publicly available pre-trained embeddings? You might have very specific data for which you have created an embedding model that you could not have found pre-trained available.
To use any embedding model that you trained yourself, you will only have to embed your documents with that model. You can then pass in the embeddings and BERTopic will do the rest:
As you can see above, we used a SentenceTransformer model to create the embedding. You could also have used 🤗 transformers, Doc2Vec, or any other embedding method.
While we are at it, why limit ourselves to transformer models? There is a reason why statistical models, such as TF-IDF, are still around. They work great out-of-the-box without much tuning!
As you might have guessed, it is also possible to use TF-IDF on the documents and use them as input for BERTopic. We simply create a TF-IDF matrix and pass it through fit_transform:
Here, you will probably notice that creating the embeddings is quite fast whereas fit_transform is quite slow. This is to be expected as reducing the dimensionality of a large sparse matrix takes some time. The inverse of using transformer embeddings is true: creating the embeddings is slow whereas fit_transform is quite fast.
If you are, like me, passionate about AI, Data Science, or Psychology, please feel free to add me on LinkedIn or follow me on Twitter.
You can find BERTopic, as well as its documentation, below: