Topic Model Visualization using pyLDAvis | by Himanshu Sharma | Towards Data Science

Creating Interactive Topic Model Visualizations
Topic Modelling is a part of Machine Learning where the automated model analyzes the text data and creates the clusters of the words from that dataset or a combination of documents. It works on finding out the topics in the text and find out the hidden patterns between words relates to those topics.
By using topic modeling we can create clusters of documents that are relevant, for example, It can be used in the recruitment industry to create clusters of jobs and job seekers that have similar skill sets. There are several ways of obtaining the topics from the model but in this article, we will talk about LDA-Latent Dirichlet Allocation.
LDA works on the matrix factorization technique in which it assumes a is a mixture of topics and it backtracks to figure what topics would have created these documents. The important part is that in this article we will create visualizations where we can analyze the clusters created by LDA.
pyLDAvis is an open-source python library that helps in analyzing and creating highly interactive visualization of the clusters created by LDA. In this article, we will see how to use LDA and pyLDAvis to create Topic Modelling Clusters visualizations.
Let’s get started...
This article will mainly focus on pyLDAvis for visualization, in order to install it we will use pip installation and the command given below will perform the installation.
pip install pyldavis
In this article, we will start by creating the model by using a predefined dataset from sklearn. In order to do all these steps, we need to import all the required libraries.
from __future__ import print_functionimport pyLDAvisimport pyLDAvis.sklearnpyLDAvis.enable_notebook()from sklearn.datasets import fetch_20newsgroupsfrom sklearn.feature_extraction.text import CountVectorizer, TfidfVectorizerfrom sklearn.decomposition import LatentDirichletAllocation
As the main focus of this article is to create visualizations you can check this link on getting a better understanding of how to create a topic model.
Now we will load the dataset that we have already imported. The newsgroup is a textual dataset so it will be helpful for this article and understanding the cluster formation using LDA.
newsgroups = fetch_20newsgroups(remove=('headers', 'footers', 'quotes'))docs_raw = newsgroups.dataprint(len(docs_raw))
Here we will see that the dataset contains 11314 rows of data. Next, we will apply CountVectorizer, TFID, etc., and create the model which we will visualize.
tf_vectorizer = CountVectorizer(strip_accents = 'unicode',                                stop_words = 'english',                                lowercase = True,                                token_pattern = r'\b[a-zA-Z]{3,}\b',                                max_df = 0.5,                                 min_df = 10)dtm_tf = tf_vectorizer.fit_transform(docs_raw)tfidf_vectorizer = TfidfVectorizer(**tf_vectorizer.get_params())dtm_tfidf = tfidf_vectorizer.fit_transform(docs_raw)
In this step, we will create the Topic Model of the current dataset so that we can visualize it using the pyLDAvis.
# for TF DTMlda_tf = LatentDirichletAllocation(n_components=20, random_state=0)lda_tf.fit(dtm_tf)# for TFIDF DTMlda_tfidf = LatentDirichletAllocation(n_components=20, random_state=0)lda_tfidf.fit(dtm_tfidf)
This is the final step where we will create the visualizations of the topic clusters. The best thing about pyLDAvis is that it is easy to use and creates visualization in a single line of code.
pyLDAvis.sklearn.prepare(lda_tf, dtm_tf, tf_vectorizer)
Check out the video below showing how interactive and visually appealing visualization is created by pyLDAvis.
Similarly, you can also create visualizations for TF-IDF vectorizer, etc. Go ahead try this and let me know your comments or any difficulty that you face in the comments section.
This post is in collaboration with Piyush Ingale.
Thanks for reading! If you want to get in touch with me, feel free to reach me at hmix13@gmail.com or my LinkedIn Profile. You can view my Github profile for different data science projects and packages tutorials. Also, feel free to explore my profile and read different articles I have written related to Data Science.