Dimensionality-Reduction with Latent Dirichlet Allocation | by Xiao Ma | Towards Data Science

Dimensionality-reduction is an unsupervised machine learning technique that is often used in conjunction with supervised models. While reducing the dimensionality often makes a feature-based model less interpretable, it’s always very effective in preventing over-fitting and shortening the training time by reducing the number of features. The most popular dimensionality-reduction techniques are no doubt the Principle Component Analysis (PCA) and its variants. However, when it comes to text data, I find Latent Dirichlet Allocation (LDA) excitingly effective.
The idea of Latent Dirichlet Allocation was first introduced in the research paper authored by David Blei, Andrew Ng, and Michael Jordan in 2003. It is described by the authors as a “generative probabilistic model of a corpus.” LDA is widely used in performing Topic Modeling — a statistical technique that can extract underlying themes/topics from a corpus.
In a traditional Bag-of-words approach for text feature extraction, we map each document directly to all the word tokens through a Document-Term matrix. This approach often results in a huge, sparse matrix with the majority of entries equal 0 — we have lots of parameters to estimate when we use such a matrix as model inputs, but many of them provide limited and sometimes “noisy” information. In the LDA approach, instead of modeling the relationships between each text document and each word token directly, the “Latent variables” are introduced as “bridges.” Each document within the corpus is characterized by a Dirichlet distribution over the latent variables (topics) and each topic is characterized by another Dirichlet Distribution over all the word tokens.
Assuming the number of documents to be N, and we choose the number of topics to be K. The idea of reducing dimension with LDA is to focus on the first Dirichlet Distribution mentioned above. For each document, we can obtain a vector of length K which represents the probability distribution of the document over K topics. Appending such vectors for all the documents, we can get an N-by-K feature matrix that can be input into our supervised model with certain labels.
Recently, inspired by the original research paper cited above, I did a small experiment in exploring the performance of LDA in dimensionality reduction. In the experiment, I was able to reduce the dimension of training data by over 99% with less than 3% drop in raw accuracy. Next, let me illustrate this experiment in greater details.
The data I used comes from the training data of the Toxic Comments Classification Kaggle challenge. Since my goal was to explore LDA in dimensionality reduction, I subset and filtered the data in the following ways to simplify the process:
The original data has 6 different labels for each comment. I only focused on the “toxic” label. In this way, I turned the problem from multi-label classification to single-label classification.Under-sampling: the original data is extremely imbalanced with more than 90% of the comments labeled as “non-toxic.” By under-sampling, I obtained a balanced dataset with 15294 comments labeled as “toxic” and another 15294 that are not labeled as “toxic.”
The original data has 6 different labels for each comment. I only focused on the “toxic” label. In this way, I turned the problem from multi-label classification to single-label classification.
Under-sampling: the original data is extremely imbalanced with more than 90% of the comments labeled as “non-toxic.” By under-sampling, I obtained a balanced dataset with 15294 comments labeled as “toxic” and another 15294 that are not labeled as “toxic.”
As the first step, I built an XG-boost classification model using the standard uni-gram approach with a TF-IDF vectorizer:
Training and testing split:
Training and testing split:
from sklearn.model_selection import train_test_splitX_train, X_test, y_train, y_test = train_test_split(data.comment_text, data.toxic, test_size = 0.25,random_state = 23)X_train.reset_index(drop = True,inplace = True)X_test.reset_index(drop = True,inplace = True)y_train.reset_index(drop = True,inplace = True)y_test.reset_index(drop = True,inplace = True)
2. Fit the TF-IDF vectorizer for n_gram models. Notice that I filter the tokens so that tokens appear in less than 15 documents more than 90% of the documents are removed. tokenizer_xm is a customized tokenizer with tokenization, lemmatization and stemming included
from sklearn.feature_extraction.text import TfidfVectorizervec_tfidf = TfidfVectorizer(ngram_range =(1,1),tokenizer=tokenizer_xm,min_df = 15, max_df = 0.9)vec_tfidf_f = vec_tfidf.fit(X_training)
3. Get the document-term matrix for n-gram model and fit with training data.
train_dtm_ngram = vec_tfidf_f.transform(X_training)from xgboost import XGBClassifierxgbc = XGBClassifier(n_estimators=200)xgbc_ngram = xgbc.fit(train_dtm_ngram,y_training)
Next, I start building the model with LDA. To use LDA, we need to obtain the Document-Term-Matrix first. Since LDA is based on raw counts, it is better to use a Count-Vectorizer instead of TF-IDF.
from sklearn.feature_extraction.text import CountVectorizervec_count = CountVectorizer(ngram_range = (1,1),tokenizer=tokenizer_xm,min_df = 15, max_df = 0.9)vec_count_f = vec_count.fit(X_train)
Create the training document-term matrix for LDA weights
vec_f = vec_count_ftrain_dtm = vec_f.transform(X_train)topic_num = 5from sklearn.decomposition import LatentDirichletAllocationlda = LatentDirichletAllocation(n_components = topic_num)lda_f = lda.fit(train_dtm)lda_weights = lda_f.transform(train_dtm)# The lda_weights is a n by k matrix where n is the number of documents and k is the number of topics# Fit the xgb-model with lda weightsfrom xgboost import XGBClassifierxgbc = XGBClassifier(n_estimators=200)xgbc_lda = xgbc.fit(lda_weights,y_train)
As in the code above, LDA requires a “n_components” parameter, which is essentially the number of topics. However, we usually don’t know how many “topics” are there in the corpus. Therefore, I repeated the process above 9 times with topic number equals to 1, 2, 3, 4, 5, 6, 12, 24, 50 and score the model against the test data, attempting to find a relatively optimal value. Below are the results.
It seems that the best topic number is 5 with an accuracy of around 84%. Below are the top 10 “salient” tokens for each of the five topics based on the Dirichlet distribution.
We are not digging deep into the performance of topic modeling with LDA in this post. But it is interesting to note the differences among topics. Topic 1 appears to be the most neutral one based while topic 5 is extremely toxic. The Dirichlet Distribution actually helps separate the non-toxic comments out from the corpus.
Next, I ran the entire process above, with 1%, 25%, 50%, 75% and 100% of the training data, recorded the performance of the N-gram based (xgbc_ngram) as well as the LDA-based (xgbc_lda) Extreme Gradient Boosting Classifier (notice that the two models are both initialized with 200 estimators and have all other hyperparameters left as defaults) and compared their precision, recall, and accuracy scores.
Below is the raw table for the plots above:
As in the figures shown above, although the accuracy for the N-gram-based model is always slightly higher than the LDA-based one, it eventually reaches 3367 distinct features with 100% of the training data while the feature number for LDA-based model still remains 5 — a significant difference! Interestingly, the LDA-based model actually out-performs the n-gram model in terms of Recall while it is not doing so well in terms of Precision. That is, the LDA-based model is better at finding all toxic examples, but not very good at preventing non-toxic comments from getting labeled as toxic. In general, the LDA-based model seems to have a better balance between precision and recall.
Using LDA for dimensionality reduction, we cannot bypass the n-gram feature extraction process. Therefore we didn’t get too much of a boost in shortening the end-to-end model building time. However, while LDA seems to be at least as efficient as PCA at dimensionality reduction, it is very interpretable. It’s a promising technique in feature-based text classification/regression models as we can compress all the information in text into a low-dimension and dense matrix so that we have the ability to add many other features to help improve the model performance without worrying about over-fitting.