Introduction to Topic Modeling using Scikit-Learn | by Ng Wai Foong | Towards Data Science

Building on top of my previous articles on natural language processing
Extractive Text Summarization Using spaCy in Python
Extract Keywords Using spaCy in Python
Let’s explore how to perform topic extraction using another popular machine learning module called scikit-learn. In this tutorial, we will be learning about the following unsupervised learning algorithms:
Non-negative matrix factorization (NMF)
Latent dirichlet allocation (LDA)
TruncatedSVD (also known as latent semantic analysis when used with count or tfidf matrices)
As for the dataset, you can choose to use your own or download the publicly available 20 News Group Dataset. It consists of approximately 20k documents related to newsgroup. There are altogether 3 variations:
20news-19997.tar.gz — contains the original unmodified 20 Newsgroups data set
20news-bydate.tar.gz — dataset is sorted by date in addition to the removal of duplicates and some headers. Split into train and test folder.
20news-18828.tar.gz — duplicates are removed and headers contain only From and Subject.
I am using 20news-18828 dataset for this tutorial. To keep things simple and short, I am going to use only 5 topics out of 20.
rec.sport.hockey
soc.religion.christian
talk.politics.mideast
comp.graphics
sci.crypt
scikit-learn’s Vectorizers expect a list as input argument with each item represent the content of a document in string. You can easily process the dataset and store it in a JSON file via the following code:
It is a lot faster to read the data from a single file before fitting it to any Vectorizers.
Please be noted that the new dataset that we have created contains unnecessary header such as From followed by email and name of a person. Such noises will impact the result of your final prediction. In an actual use case, you should remove them and retain only relevant information related to the documents.
Let’s proceed to the next section and start installing the necessary modules
It is highly recommended to create a virtual environment before you continue with the installation.
You can easily install the latest version of scikit-learn by running the following command in your terminal:
pip install -U scikit-learn
Next, continue to install pandas which is a powerful data analysis and manipulation tool.
pip install pandas
At the time of this writing, the latest numpy version is 1.19.4. It has a serious bug which causes the following error in Windows operating system.
The current Numpy installation fails to pass a sanity check due to a bug in the windows runtime
The developers behind numpy have released
“... a bugfix 1.19.3 to work around this issue. The bugfix broke something else on Linux, so we had to revert the fix in release 1.19.4, but you can still install the 1.19.3 via pip install numpy==1.19.3.”
In other words, you should install the working version based on the operating system of your machine.
Windows — 1.19.3
Linux — 1.19.4
Run the following command to install numpy. Modify the version accordingly.
pip install numpy==1.19.3
You should have the following packages installed in your virtual environment.
cycler          0.10.0joblib          1.0.0kiwisolver      1.3.1matplotlib      3.2.0numpy           1.19.3pandas          1.2.0Pillow          8.0.1pip             20.3.3pyparsing       2.4.7python-dateutil 2.8.1pytz            2020.5scikit-learn    0.24.0scipy           1.5.4setuptools      51.1.0six             1.15.0threadpoolctl   2.1.0
In this section, we are going to implement our topic modeling code using three different algorithms. Create a new Python file called test.py.
Add the following import statement at the top of the file.
import pandas as pdfrom sklearn.feature_extraction.text import TfidfVectorizer, CountVectorizerfrom sklearn.decomposition import NMF, LatentDirichletAllocation, TruncatedSVDimport numpy as npimport jsonimport random
Next, we are going to load the dataset that we have created earlier. It is a good idea to shuffle all the items in the list before you fit it to a Vectorizer.
corpus = []with open('data.json', 'r', encoding='utf8') as f:    corpus = json.loads(f.read())    random.shuffle(corpus)
Initialize the following variables which will be used later on.
n_features = 1000n_components = 5n_top_words = 20
n_components represents the number of topics while n_top_words represents the number of top words to be extracted for a single topic.
By default, there are no labels for the respective topic during predictions. You need to identify it on your own and label it manually after that.
Define the following variables which serve as the label for each algorithm:
nmf_topics = ['soc.religion.christian', 'sci.crypt', 'rec.sport.hockey', 'talk.politics.mideast', 'comp.graphics']lsa_topics = ['soc.religion.christian', 'sci.crypt', 'rec.sport.hockey', 'talk.politics.mideast', 'comp.graphics']lda_topics = ['talk.politics.mideast', 'rec.sport.hockey', 'soc.religion.christian', 'sci.crypt', 'comp.graphics']
Besides, the order and arrangement of the topics are not in order during each run especially for probabilistic model such as LDA. Hence, you should modify the arrangement after you ran the whole training and prediction pipeline.
CountVectorizer converts a collection of text documents to a matrix which contains all the token counts. Sometimes, token count is referred to as term frequency.
There are a quite useful input parameters that can be modified:
max_df — ignore terms with frequency higher than given threshold. Accepts either a float (range from 0 to 1) or integer. Float represents the proportion of documents while integer refers to absolute counts.
min_df — similar to max_df but ignore terms with frequency lower than given threshold
max_features — will only consider the given features ordered by term frequency across the entire corpus
stop_words — accepts a list of custom stopwords to be removed from the corpus. You can specify the string english which uses built-in stopwords for English.
ngram_range — a tuple which represents the lower and upper boundary for n-gram extractions.
Continue by adding the following code:
tf_vectorizer = CountVectorizer(max_df=0.95, min_df=2, max_features=n_features, stop_words='english', ngram_range=(1, 2))tf = tf_vectorizer.fit_transform(corpus)
It will remove words that occur in less than 2 documents or appear in at least 95% of the documents. Setting ngram_range to (1, 2) indicates that we are using uni-gram and bi-gram when vectorizing the dataset.
Unlike CountVectorizer, TfidfVectorizer converts documents to a matrix of TF-IDF features. The end result is akin to processing via CountVectorizer followed by TfidfTransformer.
The input parameters are more or less the same as CountVectorizer. You should this whenever possible with the only exception for algorithms that are based on word count and document count such as latent dirichlet allocation (LDA).
Append the following code which initialize the TfidfVectorizer and vectorize the input list of documents.
tfidf_vectorizer = TfidfVectorizer(max_df=0.95, min_df=2, max_features=n_features, stop_words='english', ngram_range=(1, 2))tfidf = tfidf_vectorizer.fit_transform(corpus)
Input parameters are exactly the same that we used previously for CountVectorizer.
As the name implies, this algorithm is used to find
“... two non-negative matrices (W, H) whose product approximates the non- negative matrix X.”
The results are extremely useful to reduce the dimension of features as well as source separation. Besides, it can also be used for topic extraction.
There are a few input parameters that can be fine-tuned. For example, you can change the init parameter for initializing the procedure:
random — non-negative random matrices
nndsvd — nonnegative double singular value decomposition. It is the preferred choice for sparseness.
nndsvda —nndsvd with zeros filled with the average of X. You should use when sparsity is not needed.
nndsvdar —nndsvd with zeros filled with small random values. Faster but less accurate compared to nndsvda.
By default, the value is random unless n_components < n_features. If that is the case, nndsvd will be used.
You tune the numerical solver parameter with one of the following:
cd — Coordinate Descent
mu — Multiplicative Update
NMF can be applied with three different objective functions (called beta_loss when calling the function in sklearn):
frobenius
kullback-leibler
itakura-saito — it can only be used in mu solver and the input matrix X must not contain zeros.
Objective function will be defaulted to frobenius during instantiation. The time complexity is polynomial when running NMF.
Initialize a new instance which takes in the output matrix of TfidfVectorizer.
nmf = NMF(n_components=n_components, random_state=1, alpha=.1, l1_ratio=.5, init='nndsvd').fit(tfidf)
alpha and l1_ratio are related to regularization. Set alpha value to 0 if you prefer to have no regularization.
LDA is a good generative probabilistic model for identifying abstract topics from discrete dataset such as text corpora.
LDA in scikit-learn is based on online variational Bayes algorithm which supports the following learning_method:
batch — use all training data in each update.
online — use mini-batch of the training data for each update. Will be a lot faster than batch if the training data size is large.
From version 0.20 onwards, the default learning_method is batch.
You should use CountVectorizer when fitting LDA instead of TfidfVectorizer since LDA is based on term count and document count. Fitting LDA with TfidfVectorizer will result in rare words being dis-proportionally sampled. As a result, they will have greater impact and influence on the final topic distribution.
Use the following code to create a new instance of LDA.
lda = LatentDirichletAllocation(n_components=n_components, random_state=1).fit(tf)
Since LDA is a probabilistic model, you will get some differences in the end result each time you run it. Hence, it is a good idea to set the random_state parameter to a fixed number and save the model locally using pickle to preserve how it infers the topics later on.
In addition, scikit-learn also comes with a great and useful dimensionality reduction model called Truncated Singular Value Decomposition (TruncatedSVD). Unlike PCA, this model computes the singular value decomposition without centering the data.
In the event where TruncatedSVD model is fitted with count or tfidf matrices, it is also known as Latent Semantic Analysis (LSA). Most articles tend to refer this as LSA instead of TruncatedSVD. For consistency, I will refer it as LSA as well in this tutorial.
It supports two different algorithms:
randomized — a fast randomized SVD solver
arpack — based on SciPy’s ARPACK wrapper as eigensolver
Create a new instance of TruncatedSVD which fits against tfidf’s matrix via the following code.
lsa = TruncatedSVD(n_components=n_components, random_state=1, algorithm='arpack').fit(tfidf)
Please be noted that you should fit instance of this class once with pre-defined random_state before performing transformations. This prevents sign indeterminnancy issue, as the output from transform depends on the algorithm as well as random_state.
Once you are done with it, let’s create a generic function which returns topics from a model as DataFrame. Our implementation is based on the print_top_words function from the following tutorial but with a little twist to it. Feel free to modify it based on your use cases.
We will need another function to do inference for any input text of our choice. Define a new function called get_inference and add the following code inside it.
This is my own implementation which returns
label of the topic with the highest confidence
confidence value for all the topics
labels of all the topics that exceed a certain threshold
Please be noted that confidence value for a particular topic can be a negative number depending on the algorithm used.
The last step is to run both of the functions that we have created earlier and print out the results. I am going to use the following input text in which should be mapped to comp.graphics.
text = 'you should use either jpeg or png files for it'
Continue by calling the functions on all of the models that we have.
For NMF, I got the following output which illustrated the topics distribution and final inference of the input text:
The result for LSA is a little less promising even though the final result is accurate. There are quite a few of words that are modeled wrongly based on the topics distribution.
On the other hand, LDA returned the following result. The word edu was predicted as the top word for three topics. This is mainly because the dataset that I used has From headers which contains quite a number of email addresses ends with .edu.
Please be noted that the order of topics and top words might be different on each run especially if you did not specify a random_state or uses online learning.
Consider saving both the Vectorizer and model locally using pickle if you intend to preserve the same results on each execution.
Let’s recap what we have learned today.
We started off with exploring the 20 NewGroup dataset which categorized into three different variations. We also processed the data and used only 5 topics instead of the full dataset.
Then, we moved on to install the necessary Python modules and loaded our dataset in a Python file.
Furthermore, we created both the CountVectorizer and TfidfVectorizer which are used to vectorize the documents before fitting them to our models.
We built three different models based on non-negative matrix factorization, latent dirichlet allocation and TruncatedSVD.
Lastly, we created generic functions to get topics and perform inference from the models that we have built.
Thanks for reading this piece. Hope to see you again in the next article!
20 News Group DatasetStackOverflow — Numpy runtime issueSkLearn — Topic Extraction
20 News Group Dataset
StackOverflow — Numpy runtime issue
SkLearn — Topic Extraction