Tweet Topic Modeling: Using Short Text Topic Modeling on Tweets
By: John Bica
Tags: Tweet, Python, Clustering, NLP, Topic Modeling

PART 3: Applying Short Text Topic Modeling (STTM)

In the previous article, we had preprocessed all of our tweets producing a data frame with the lemmatized tokens for each tweet in our data set. The final csv from that is provided here for your reference to follow along if you are just joining us in part 3.

This article will walk through performing some initial exploratory analysis and then building a STTM model pipeline using the GSDMM algorithm on our data to generate topics. We will then interpret our results and attempt to assign topics based on the word distribution in each topic.

Exploratory Data Analysis

Let’s start by doing some EDA. Since one of our key objectives with the health news tweets data set is to explore trends over time, it may be useful to get an idea of the density of daily tweet counts.

import pandas as pd

import matplotlib.pyplot as plt

tweets_df = pd.read_csv(r'data/preprocessed_tweets.csv') #load data # convert datetime to date and add year column

tweets_df['date'] = pd.to_datetime(tweets_df['date'],

errors='coerce')

tweets_df['year'] = tweets_df['date'].dt.year

tweets_df['date'] = tweets_df['date'].dt.date # plot tweet daily counts over time

tweets_df.groupby('date')['tweet'].count().plot(kind='line', color='green', figsize=(16,8))

plt.xlabel('Year')

plt.ylabel('Daily Tweets')

plt.show()

It looks like there is a huge spike in daily tweets towards the end of our dataset timeline in 2020. This actually makes sense for a dataset containing health news tweets since this was around the start of the 2020 COVID-19 health pandemic 👨‍⚕️⚕️️.

Another area we may want to take a look at is the average number of words remaining (after we’ve taken out stop words of course) in a tweet across our different news sources.

# calculate number of tokens for each tweet

tweets_df[‘ntokens’] = tweets_df[‘tokens’].str.split().str.len() # plot average word count by user

tweets_df.groupby(‘username’)

[‘ntokens’].mean().plot(kind=’barh’,figsize=(10,5))

plt.xlabel(‘Average Words’)

plt.ylabel(‘News Account’)

plt.show()

Looks like most news sources have about 6–8 tokens on average. I don’t know about you but that sounds like an A+ example of short text to me!

STTM Overview

Before we start to implement an STTM approach, let’s talk a bit more about why we are using this over a traditional topic modeling algorithm such as LDA. From a very high level, LDA produces 2 statistical distributions:

1) a set of word distributions for each topic 2) a set of topic distributions for each document

What this means is that each topic generated has associated probabilities that words strongly relate to it and that a document could be composed of a mixture of topics, each with their own weighting. For example, if your document is a news article 📰 about stars⭐ and spherical-shaped planets 🌎, LDA might generate the following distribution of topics and words:

Simplistic but reasonable example of what a topic/word distribution may look like for a science news article (Image by author)

This works well on medium or large sized texts but it doesn’t always perform as well on short texts (< 50 words) because of these key characteristics:

Each short text lacks enough word co-occurrence information. Word co-occurrence is the chance that two words occur together in a text and it helps with inferring the word distribution of topics generated.

is the chance that two words occur together in a text and it helps with inferring the word distribution of topics generated. Most topics are probably generated by only 1 topic due to containing minimal words.

Statistical information of words among texts cannot fully capture words that are semantically related but rarely co-occur.

Therefore, most STTM techniques make the initial assumption that a short text comes from only one topic, reducing the overlapping topics that we see with LDA.

Gibbs Sampling Dirichlet Mixture Model

The specific model we will use for STTM is Gibbs Sampling Dirichlet Mixture Model (GSDMM) which is a modified LDA algorithm that uses the simple assumption of one topic assigned to one text.

Before diving into its code, let’s understand what it does from a high level using an analogy called the Movie Group Process (MGP) presented by the researchers who introduced it.

The idea is simple. Imagine a professor is leading a film class of students. At the start of class, students are all asked to write down their favorite movies (relatively shortlist). The students represent documents and their movie lists represent words. Next, the students are randomly assigned to K tables. The goal is to cluster and group them in a way that students within the same table share similar movie interests. Lastly, the professor repeatedly reads the class roster; each time a student’s name is called, they must select a new table satisfying one or both of the following conditions:

1 — Choose a table with more students than their current table.

2 — Choose a table where students share similar movie interests.

Condition 1 improves completeness, all students with similar movie interests are at the same table rather than spread across different tables. Condition 2 helps lead to better homogeneity, ensuring that only members sharing similar interests are at the table. After satisfying these conditions and repeating them consistently until we near optimality (convergence), we expect some tables to disappear while others grow. The hope is that students will eventually arrive at an optimal table configuration. Simply, this is what the GSDMM algorithm does!

Visual of Movie Group Process analogy (Image by author)

STTM Implementation

Now we will begin to build our STTM pipeline and run it on our health news tweet data set. If using your own dataset, this part assumes you have preprocessed and tokenized your tweets. Otherwise, you can refer to part 2.

The first thing we will do is load in the GSDMM model from GitHub into a folder named gsdmm . Run the following command in a terminal.

git clone https://github.com/rwalk/gsdmm.git gsdmm

Next, we will import the model. Using the data frame we loaded in earlier during EDA, we will convert our tokens into a list and then create a final list of lists to hold all the tokens across all of our tweets. This is necessary as it is the proper input format for the model.

import numpy as np

import pandas as pd

import pickle

import gsdmm

from gsdmm import MovieGroupProcess

from tqdm import tqdm # convert string of tokens into tokens list

tweets_df['tokens'] = tweets_df.tokens.apply(lambda x:

re.split('\s', x)) # create list of token lists

docs = tweets_df['tokens'].tolist()

Output sample for docs — input required for gsdmm model

Before running the model, we will need to understand and choose the hyper-parameters for GSDMM:

K — Represents the number of maximum topics that will be found by the model. Recall, that GSDMM will empty useless clusters and eventually attempt to find the “true” number of topics. This works in theory but highly depends on your dataset and the other hyperparameters used. For example, if you run it with K set to 300, you may find yourself getting 300 topics which may or may not be realistic or useful for your data. In our case, considering how short each tweet is and the fact that all are already under the health topic domain, I have chosen K=10 since we are looking to discover sub-topics within health. For further reading, there are several different ways to choose an ideal K that I include in references at the end.

— Represents the number of maximum topics that will be found by the model. Recall, that GSDMM will empty useless clusters and eventually attempt to find the “true” number of topics. This works in theory but highly depends on your dataset and the other hyperparameters used. For example, if you run it with K set to 300, you may find yourself getting 300 topics which may or may not be realistic or useful for your data. In our case, considering how short each tweet is and the fact that all are already under the health topic domain, I have chosen since we are looking to discover sub-topics within health. For further reading, there are several different ways to choose an ideal K that I include in references at the end. Alpha α — In regards to the MGP, alpha controls a factor that decides how easily a table gets removed when it’s empty (low alpha = fewer tables). As a result, the number of non-empty clusters will get larger as alpha increases and there will be more clusters with only one or few documents. By default, a value of α=0.1 works fine on most datasets.

— In regards to the MGP, alpha controls a factor that decides how easily a table gets removed when it’s empty (low alpha = fewer tables). As a result, the number of non-empty clusters will get larger as alpha increases and there will be more clusters with only one or few documents. By default, a value of works fine on most datasets. Beta β — In regards to the MGP, beta controls how a table is chosen for a student based on similarity to the table rather than the popularity of the table (low beta = more similar clusters). As a result, the number of non-empty clusters will get smaller as beta increases since the model will give more emphasis on selecting popular clusters rather than similar clusters. By default, a value of β=0.1 works fine on most data sets.

— In regards to the MGP, beta controls how a table is chosen for a student based on similarity to the table rather than the popularity of the table (low beta = more similar clusters). As a result, the number of non-empty clusters will get smaller as beta increases since the model will give more emphasis on selecting popular clusters rather than similar clusters. By default, a value of works fine on most data sets. N_iters — In regards to the MGP, this represents the number of iterations or number of times that each student is reassigned to a new table by the professor. According to the original research paper, GSDMM converges fairly quickly and stability is generally reached within 10 though we will use n_iters=30 as our default.

With our hyperparameters configured, we are ready to train the model! ✊

# train STMM model

mgp = MovieGroupProcess(K=10, alpha=0.1, beta=0.1, n_iters=30)

vocab = set(x for doc in docs for x in doc)

n_terms = len(vocab)

y = mgp.fit(docs, n_terms) # save model

with open(‘dumps/trained_models/10clusters.model’, ‘wb’) as f:

pickle.dump(mgp, f)

f.close()

Once model training is done (note: training takes some time depending on your dataset and iterations), we will want to explore the topics found and assess them. First, let’s define some helper functions that will make that easier.

The following code will provide us with statistical insight into the number of tweets in each cluster and their top word distribution.

doc_count = np.array(mgp.cluster_doc_count)

print(‘Number of documents per topic :’, doc_count)

print(‘*’*20) # topics sorted by the number of documents they are allocated to

top_index = doc_count.argsort()[-10:][::-1]

print(‘Most important clusters (by number of docs inside):’,

top_index)

print(‘*’*20) # show the top 5 words in term frequency for each cluster

topic_indices = np.arange(start=0, stop=len(doc_count), step=1)

top_words(mgp.cluster_word_distribution, topic_indices, 5)

The results look pretty good at first glance! Here are some general patterns we can infer from some of the topics:

Cluster 0 appears to involve healthcare and the politics behind it

appears to involve healthcare and the politics behind it Cluster 1 appears to focus solely on viruses and outbreaks

appears to focus solely on viruses and outbreaks Cluster 5 appears to relate to healthy lifestyle and dieting

appears to relate to healthy lifestyle and dieting Cluster 7 appears to involve the ongoing abortion rights discussion

appears to involve the ongoing abortion rights discussion Cluster 8 appears to focus on cigarettes and vaping health effects

If we want to dig into seeing how important certain words are to a particular cluster, we can use the cluster_importance function defined earlier to create a topic — word matrix, where each value represents the word importance for that particular topic.

phi = cluster_importance(mgp) # initialize phi matrix

phi[1][‘coronavirus’]

Output: 0.015673293786310202

phi[0][‘coronavirus’]

Output: 0.0023181196054210713

Here, we can see that the word ‘coronavirus’ has a value of 0.016 for cluster 1 (which appears to be about viruses) but a very small value relatively for cluster 0 (which appears to be about healthcare and politics).

Once we’re done exploring and are comfortable and satisfied with the cluster results, the next step is to assign actual human interpretable topics to each cluster. Remember, topic modeling generates “latent” topics. We still need a human at the end of the process to interpret the clusters and define what each one is about. Doing so is quite an art 🎨 though.

Let’s try our best for our 10 clusters!

topic_dict = {}

topic_names = [‘healthcare & policy’,

‘virus/outbreaks’,

‘cancer studies affecting woman/babies’,

‘miscellaneous studies affecting women/children’,

‘cancer & heart disease’,

‘diet & exercise’,

‘health & medical workers’,

‘abortion’,

‘vaping & cigarettes’,

‘drug costs & opioid crisis’] for i, topic_num in enumerate(topic_indices):

topic_dict[topic_num]=topic_names[i]

With the topics defined, we can now use our topic_allocation function to assign a topic to each tweet.

topic_allocation(tweets_df, docs, mgp, topic_dict)

Note: this custom topic allocation function is built upon the original choose_best_label function available in the gsdmm module, which selects the topic with the highest probability of belonging to a document. If you’re looking to use a minimum threshold when allocating topics in order to assign a ‘Others’ topic for any tweets that don’t meet the set threshold, you can add some code to the helper function to take in an additional threshold input parameter. You’ll need to adjust the code to check for this condition before assigning a topic.

Let’s take a look at a sample of our tweets and their assigned topics.