Twitter Topic Modeling. Using Machine Learning (Gensim Linear... | by Amin Azad | Towards Data Science

I’m a machine learning geek and I’d like to apply machine learning on everything that I can just to see the results. On the other hand, I started a new role at SoMe (Social Media Management Platform) and within the data science team, we continuously brainstorm around using machine learning to create more value for our users and help them grow their follower base. We thought long and hard to come up with the best ideas, and in the process, we realized the first step is to provide the user with an overall understanding of their follower base.
There is a wide range of data science and machine learning techniques that could be applied for providing a better understanding of your followers, from applying CNN (Convolutional Neural Networks) on Instagram to applying natural language processing techniques on Twitter, Linkedin or any other text-based data. Majority of our users use SoMe to sort and schedule their future posts on Twitter so as a first step we decided to provide our users with insights about the topics their Twitter followers engage with the most. To do that we first had to define and score follower engagement. After a long discussion, we defined and scored follower engagement as “When a follower interacts with the content that user posts by 1. Retweeting (5/5) — 2. Retweeting with a comment (4/5) — 3. Liking (3/5) —4. Comment(2/5) — 5. Mentioning (1/5)”.
Next step was to actually get the data from a Twitter api, I will write about applying for a Twitter api and using Tweepy and other tools to pull data and refine data to get to the data you need from the api in a future post. You can use the following techniques on any kind of text data and figure out what are the most important topics discussed within your dataset.
Depending on your choice of python notebook, you are going to need to install and load the following packages to perform topic modeling. Among the SoMe, data science team we use a wide range of notebook options, from Azure to Jupyter labs and notebook. My personal favorite however is Google Colab. I recommend running all the following snippets on either Google Colab or Jupyter notebook. I’m also going to link the Github repository for this project and a link to the final notebook we used for your reference.
# Installations
import sys
if 'google.colab' in sys.modules:
    !pip install emoji --upgrade
    !pip install pandas-profiling==2.*
    !pip install plotly==4.*
    !python -m spacy download en_core_web_lg
    !pip install pyldavis
    !pip install gensim
    !pip install chart_studio
    !pip install --upgrade autopep8

# Required Libraries

#Base and Cleaning 
import json
import requests
import pandas as pd
import numpy as np
import emoji
import regex
import re
import string
from collections import Counter

#Visualizations
import plotly.express as px
import seaborn as sns
import matplotlib.pyplot as plt 
import pyLDAvis.gensim
import chart_studio
import chart_studio.plotly as py 
import chart_studio.tools as tls

#Natural Language Processing (NLP)
import spacy
import gensim
from spacy.tokenizer import Tokenizer
from gensim.corpora import Dictionary
from gensim.models.ldamulticore import LdaMulticore
from gensim.models.coherencemodel import CoherenceModel
from gensim.parsing.preprocessing import STOPWORDS as SW
from sklearn.decomposition import LatentDirichletAllocation, TruncatedSVD
from sklearn.feature_extraction.text import CountVectorizer, TfidfVectorizer
from sklearn.model_selection import GridSearchCV
from pprint import pprint
from wordcloud import STOPWORDS
stopwords = set(STOPWORDS)

After pulling and refining the data from the Twitter api and importing the required packages, we had to clean the data from emojis and urls so we can tokenize it for the next steps. You can find the cleaning syntaxes we used in the below snippet:
def give_emoji_free_text(text):
    """
    Removes emoji's from tweets
    Accepts:
        Text (tweets)
    Returns:
        Text (emoji free tweets)
    """
    emoji_list = [c for c in text if c in emoji.UNICODE_EMOJI]
    clean_text = ' '.join([str for str in text.split() if not any(i in str for i in emoji_list)])
    return clean_text

def url_free_text(text):
    '''
    Cleans text from urls
    '''
    text = re.sub(r'http\S+', '', text)
    return text

# Apply the function above and get tweets free of emoji's
call_emoji_free = lambda x: give_emoji_free_text(x)

# Apply `call_emoji_free` which calls the function to remove all emoji's
df['emoji_free_tweets'] = df['original_tweets'].apply(call_emoji_free)

#Create a new column with url free tweets
df['url_free_tweets'] = df['emoji_free_tweets'].apply(url_free_text)

As you can see in the table below, there are a lot of emojis and urls you want to remove before tokenizing. Data cleaning is essential for ultimately delivering accurate results, we do not want words from websites or emojis end up in our topic modeling results since they deliver almost no value in figuring out what is the general topic of a bag of words.
Our goal in the data pre-processing stage is to convert sentences into words, convert words to their root and removing words that are too common or too irrelevant to the purpose of our topic modeling project. We used the following techniques to reach our goal, I’m going to share the code and walk you through each stage:
Tokenization: Split the text into sentences and the sentences into words. Lowercase the words and remove punctuation.
Words that have fewer than 3 characters are removed.
All stopwords are removed.
Words are lemmatized and stemmed — words in the third person are changed to first person and verbs in past and future tenses are changed into the present and are reduced to their root form.
Tokenization is always the first step before we can do any text data processing. What this means is that spaCy will segment sentences into words, punctuations, symbols and others by applying specific rules to each language. Spacy is a pre-trained natural language processing model capable of figuring out the relationship between words. You can learn more about Spacy here.
# Load spacy
# Make sure to restart the runtime after running installations and libraries tab
nlp = spacy.load('en_core_web_lg')

# Tokenizer
tokenizer = Tokenizer(nlp.vocab)


# Custom stopwords
custom_stopwords = ['hi','\n','\n\n', '&amp;', ' ', '.', '-', 'got', "it's", 'it’s', "i'm", 'i’m', 'im', 'want', 'like', '$', '@']

# Customize stop words by adding to the default list
STOP_WORDS = nlp.Defaults.stop_words.union(custom_stopwords)

# ALL_STOP_WORDS = spacy + gensim + wordcloud
ALL_STOP_WORDS = STOP_WORDS.union(SW).union(stopwords)


tokens = []

for doc in tokenizer.pipe(df['url_free_tweets'], batch_size=500):
    doc_tokens = []    
    for token in doc: 
        if token.text.lower() not in STOP_WORDS:
            doc_tokens.append(token.text.lower())   
    tokens.append(doc_tokens)

# Makes tokens column
df['tokens'] = tokens

Lemmatization is a process where we convert words to its root word. For example: ‘Studying’ becomes ‘Study’, ‘Meeting becomes ‘Meet’, ‘Better’ and ‘Best’ become ‘Good’.The advantage of this is, we get to reduce the total number of unique words in the dictionary. As a result, the number of columns in the document-word matrix will be denser with lesser columns. The ultimate goal of lemmatization is to help the LDA model to produce better topics in the end.
# Make tokens a string again
df['tokens_back_to_text'] = [' '.join(map(str, l)) for l in df['tokens']]

def get_lemmas(text):
    '''Used to lemmatize the processed tweets'''
    lemmas = []
    
    doc = nlp(text)
    
    # Something goes here :P
    for token in doc: 
        if ((token.is_stop == False) and (token.is_punct == False)) and (token.pos_ != 'PRON'):
            lemmas.append(token.lemma_)
    
    return lemmas

df['lemmas'] = df['tokens_back_to_text'].apply(get_lemmas)

# Make lemmas a string again
df['lemmas_back_to_text'] = [' '.join(map(str, l)) for l in df['lemmas']]

# Tokenizer function
def tokenize(text):
    """
    Parses a string into a list of semantic units (words)
    Args:
        text (str): The string that the function will tokenize.
    Returns:
        list: tokens parsed out
    """
    # Removing url's
    pattern = r"http\S+"
    
    tokens = re.sub(pattern, "", text) # https://www.youtube.com/watch?v=O2onA4r5UaY
    tokens = re.sub('[^a-zA-Z 0-9]', '', text)
    tokens = re.sub('[%s]' % re.escape(string.punctuation), '', text) # Remove punctuation
    tokens = re.sub('\w*\d\w*', '', text) # Remove words containing numbers
    tokens = re.sub('@*!*\$*', '', text) # Remove @ ! $
    tokens = tokens.strip(',') # TESTING THIS LINE
    tokens = tokens.strip('?') # TESTING THIS LINE
    tokens = tokens.strip('!') # TESTING THIS LINE
    tokens = tokens.strip("'") # TESTING THIS LINE
    tokens = tokens.strip(".") # TESTING THIS LINE

    tokens = tokens.lower().split() # Make text lowercase and split it
    
    return tokens

# Apply tokenizer
df['lemma_tokens'] = df['lemmas_back_to_text'].apply(tokenize)

We produce a base model first to be used to track our progress as we go through the hyper-parameter tuning stage. The LDA topic model algorithm requires a document word matrix and a dictionary as the main inputs. For the first few steps to be taken before running the LDA model, we created a dictionary, filtered the extremes and, create a corpus object which is the document matrix LDA model needs as the main input.
We prepared everything required to train the LDA model. In addition to the corpus and dictionary, we needed to provide the number of topics as well. We chose 5 for the base model. In the hyperparameter tuning stage we will reach to an optimum number of topics to be used.
By printing out the topics produced from the LDA model, we can have a rough guess about the topic related to each bag of words. It’s important to mention the bag of words have been sorted from the most relevant to the least relevant for each topic.
Model perplexity and topic coherence provide a convenient measure to judge how good a given topic model is. In my experience, topic coherence score, in particular, has been more helpful.
We got 0.17 for coherence score, which is a very low score for almost any LDA model but remember this is only the base model, we are going to substantially improve it.
Now that the LDA model is built, the next step is to examine the produced topics and the associated keywords. There is no better tool than pyLDAvis package’s interactive chart and is designed to work well with Google Colab and Jupyter notebooks.
So how to infer pyLDAvis’s output?
Each bubble on the left-hand side plot represents a topic. The larger the bubble, the more prevalent is that topic. A good topic model will have fairly big, non-overlapping bubbles scattered throughout the chart instead of being clustered in one quadrant. A model with too many topics, will typically have many overlaps, small sized bubbles clustered in one region of the chart.
Alright, if we move the cursor over one of the bubbles, the words and bars on the right-hand side will update. These words are the salient keywords that form the selected topic. We have successfully built a good looking topic model. Up next, we will improve upon this model (Hyperparameter Tuning) by using Scikit-learn’s Grid Search and then we will focus on how to arrive at the optimal number of topics and other variables in within the LDA model.
The most important tuning parameter for LDA models is n_components (number of topics). In addition, we are going to search learning_decay (which controls the learning rate) as well. Besides these, other possible search params could be learning_offset (down weight early iterations. Should be > 1) and max_iter. These could be worth experimenting if you have enough time and computing resources.
Be warned, the grid search constructs multiple LDA models for all possible combinations of param values in the param_grid dict. So, this process can consume a lot of time and resources.
We know there is a high chance number of topics is much more than 10 but through the grid search we found out 10 topics performs better than other number of topics. This makes us think deeply about the results, and we figured Scikit-learn’s grid search tracks perplexity not coherence value and for our use case, coherence value delivers the best result. In the next step we’re going to get to an optimal number of topics.
Our approach to finding the optimal number of topics was to build many LDA models with different values of number of topics and pick the one that gives the highest coherence value.
Choosing a number of topic that marks the end of a rapid growth of topic coherence usually offers meaningful and interpretable topics. Picking an even higher value can sometimes provide more granular sub-topics.If you see the same keywords being repeated in multiple topics, it’s probably a sign that the ‘number of topics’ is too large. By following these principles we chose 68 topics as an optimal number of topics for our use case.
Passes, chunksize and update_every are parameters that have a EM/Variational relationship together. We are not going to go into the details of EM/Variational Bayes here, but if you are curious, check out this google forum post and the paper it references here. For our use case, hyperparameter tuning number of passes would suffice considering chuncksize is not substantial and update_every would not change much in the final results.
We tried many numbers of passes and 20 seems to produce the best result. Not substantially though, only a few decimal points.
Alpha is the hyperparameter for the Dirichlet prior. The Dirichlet prior is the distribution from which we draw theta. And theta becomes the parameter that decides what shape the topic distribution is. So essentially, alpha influences how we draw topic distributions. That is why we’re gonna try to hyperparameter tune to choose the best alpha type that gives you a better topic distribution.
In our use case, symmetric alpha delivers a better result, not substantially but we are going to use symmetric alpha for our LDA model.
Learning decay controls the learning rate of the model. Since you can only choose 0.5, 0.7 and 0.9, we’re going to try all three and see which option delivers the best coherence value.
In our use case, 0.5 decay delivers the best coherence value. In fact, decay and coherence value in our use case seems to be negatively correlated.
iterations is somewhat technical, but essentially it controls how often we repeat a particular loop over each document. It is important to set the number of “iterations” high enough.
In our use case, we tried 50 to 100 iterations but since we want to prevent the model from overfitting we chose 70 which showed a decent amount of coherence score improvement.
This hyperparameter disregards topics with probability assigned lower than what is assigned. It can not be lower than 0.01 and higher than 0.1.
We tried the full range of 0.01 to 0.1 and coherence value remained constant across the full range so we left it to the default value of 0.01 to get the most amount of topics possible.
It is always a good practice to track your progress from the beginning. We have been tracking our progress going through 24 iterations of model hyperparameter tuning and our best model delivers a coherence value of 0.47 which is a pretty good number considering the wide range of topics that could be discussed among users follower base.
We finally got to our final model. We’re currently using the model in the snippet below at SoMe to provide our users with the topics their followers interact with the most.
If we take a closer look at the topics produced by the final model, we can guess with a high accuracy what each topic is about. Now we have to remember they are sorted from the highest importance to the least. For example, if we look at Topic one we figure it's about the COVID-19 crisis as the first three words are “die-virus-lockdown” and not so much about “gold” since “gold” is the fourth word which does not reflect a high probability in the bag of words.
As we covered before, a good topic model will have non-overlapping, fairly big sized blobs for each topic and if we increase number of topics based on the use case we’re going to have smaller blobs and some overlap in some cases but overall we can see in the final model, the distribution and overlap have significantly improved. This seems to be the case here. So, we are good.
We’ve covered some cutting-edge topic modeling approaches in this post. We continuously work to build new features for our users and harness machine learning’s power to provide a better experience for our users. To improve this model you can explore modifying it by using gensim LDA Mallet which in some cases provides more accurate results. For those concerned about the time, memory consumption and variety of topics when building topic models check out the gensim tutorial on LDA.
At the end, I have to thank the rest of SoMe data science team especially Jacob Padgett and Lawrence Kimsey. At SoMe, we strongly believe in open source software and you can find all our analytics models notebooks here.