Topic Modeling Articles with NMF. Extracting topics is a good... | by Rob Salgado | Towards Data Science

Extracting topics is a good unsupervised data-mining technique to discover the underlying relationships between texts. There are many different approaches with the most popular probably being LDA but I’m going to focus on NMF. I’ve had better success with it and it’s also generally more scalable than LDA.
This article covers how to:
Prepare the text for topic modeling
Extract topics from articles
Summarize those topics
Find the best number of topics to use for the model automatically
Find the highest quality topics among all the topics
Predict the topic of a new article
As always, all the code and data can be found in a repository on my GitHub page.
I’m using full text articles from the ‘Business’ section of CNN. The articles appeared on that page from late March 2020 to early April 2020 and were scraped. The scraper was run once a day at 8 am and the scraper is included in the repository. The articles on the ‘Business’ page focus on a few different themes including investing, banking, success, video games, tech, markets etc.
Let’s do some quick exploratory data analysis to get familiar with the data. There are 301 articles in total with an average word count of 732 and a standard deviation of 363 words. Here are the first five rows.
In terms of the distribution of the word counts, it’s skewed a little positive but overall it’s a pretty normal distribution with the 25th percentile at 473 words and the 75th percentile at 966 words. There are about 4 outliers (1.5x above the 75th percentile) with the longest article having 2.5K words.
Here are the top 20 words by frequency among all the articles after processing the text. ‘Company’, ‘business’, ‘people’, ‘work’ and ‘coronavirus’ are the top 5 which makes sense given the focus of the page and the time frame for when the data was scraped.
Non-Negative Matrix Factorization (NMF) is an unsupervised technique so there are no labeling of topics that the model will be trained on. The way it works is that, NMF decomposes (or factorizes) high-dimensional vectors into a lower-dimensional representation. These lower-dimensional vectors are non-negative which also means their coefficients are non-negative.
Using the original matrix (A), NMF will give you two matrices (W and H). W is the topics it found and H is the coefficients (weights) for those topics. In other words, A is articles by words (original), H is articles by topics and W is topics by words.
So assuming 301 articles, 5000 words and 30 topics we would get the following 3 matrices:
A = tfidf_vectorizer.transform(texts)W = nmf.components_H = nmf.transform(A)A = 301 x 5000W = 30 x 5000H = 301 x 30
NMF will modify the initial values of W and H so that the product approaches A until either the approximation error converges or the max iterations are reached.
In our case, the high-dimensional vectors are going to be tf-idf weights but it can be really anything including word vectors or a simple raw count of the words.
This is one of the most crucial steps in the process. As the old adage goes, ‘garbage in, garbage out’. When dealing with text as our features, it’s really critical to try and reduce the number of unique words (i.e. features) since there are going to be a lot. This is our first defense against too many features.
The scraped data is really clean (kudos to CNN for having good html, not always the case). You should always go through the text manually though and make sure there’s no errant html or newline characters etc. which can definitely show up and hurt the model.
Below is the function I’m using which:
tokenizes the text
lower cases the text
expands out contractions
stems the text
removes punctuation, stop words, numbers, single characters and words with extra spaces (artifact from expanding out contractions)
This is kind of the default I use for articles when starting out (and works well in this case) but I recommend modifying this to your own dataset. For example I added in some dataset specific stop words like ‘cnn‘ and ‘ad’ so you should always go through and look for stuff like that. These are words that appear frequently and will most likely not add to the model’s ability to interpret topics.
Here’s an example of the text before and after processing:
‘In the new system “Canton becomes Guangzhou and Tientsin becomes Tianjin.” Most importantly, the newspaper would now refer to the country’s capital as Beijing, not Peking. This was a step too far for some American publications. In an article on Pinyin around this time, the Chicago Tribune said that while it would be adopting the system for most Chinese words, some names had “become so ingrained’
‘new canton becom guangzhou tientsin becom tianjin import newspap refer countri capit beij peke step far american public articl pinyin time chicago tribun adopt chines word becom ingrain‘
Now that the text is processed we can use it to create features by turning them into numbers. There’s a few different ways to do it but in general I’ve found creating tf-idf weights out of the text works well and is computationally not very expensive (i.e runs fast).
You can read more about tf-idf here. Some other feature creation techniques for text are bag-of-words and word vectors so feel free to explore both of those.
For feature selection, we will set the ‘min_df’ to 3 which will tell the model to ignore words that appear in less than 3 of the articles. We’ll set the ‘max_df’ to .85 which will tell the model to ignore words that appear in more than 85% of the articles. This will help us eliminate words that don’t contribute positively to the model.
After processing we have a little over 9K unique words so we’ll set the max_features to only include the top 5K by term frequency across the articles for further feature reduction.
Besides just the tf-idf wights of single words, we can create tf-idf weights for n-grams (bigrams, trigrams etc.). To do that we’ll set the n_gram range to (1, 2) which will include unigrams and bigrams.
We also need to use a preprocesser to join the tokenized words as the model will tokenize everything by default.
texts = df['processed_text']tfidf_vectorizer = TfidfVectorizer(    min_df=3,    max_df=0.85,    max_features=5000,    ngram_range=(1, 2),    preprocessor=' '.join)tfidf = tfidf_vectorizer.fit_transform(texts)
Now that we have the features we can create a topic model. 👍
First here is an example of a topic model where we manually select the number of topics. After I will show how to automatically select the best number of topics. The hard work is already done at this point so all we need to do is run the model.
nmf = NMF(    n_components=20,    init='nndsvd').fit(tfidf)
The only parameter that is required is the number of components i.e. the number of topics we want. This is the most crucial step in the whole topic modeling process and will greatly affect how good your final topics are. For now we will just set it to 20 and later on we will use the coherence score to select the best number of topics automatically.
I’m also initializing the model with ‘nndsvd’ which works best on sparse data like we have here. Everything else we’ll leave as the default which works well. However, feel free to experiment with different parameters.
To evaluate the best number of topics, we can use the coherence score. Explaining how it’s calculated is beyond the scope of this article but in general it measures the relative distance between words within a topic. Here is the original paper for how it’s implemented in gensim.
There are a few different types of coherence score with the two most popular being c_v and u_mass. c_v is more accurate while u_mass is faster. I’ll be using c_v here which ranges from 0 to 1 with 1 being perfectly coherent topics.
I like sklearn’s implementation of NMF because it can use tf-idf weights which I’ve found to work better as opposed to just the raw counts of words which gensim’s implementation is only able to use (as far as I am aware). However, sklearn’s NMF implementation does not have a coherence score and I have not been able to find an example of how to calculate it manually using c_v (there is this one which uses TC-W2V). If anyone does know of an example please let me know!
Therefore, we’ll use gensim to get the best number of topics with the coherence score and then use that number of topics for the sklearn implementation of NMF.
Obviously having a way to automatically select the best number of topics is pretty critical, especially if this is going into production. Using the coherence score we can run the model for different numbers of topics and then use the one with the highest coherence score. This certainly isn’t perfect but it generally works pretty well.
For the number of topics to try out, I chose a range of 5 to 75 with a step of 5. This just comes from some trial and error, the number of articles and average length of the articles. Each dataset is different so you’ll have to do a couple manual runs to figure out the range of topic numbers you want to search through. Running too many topics will take a long time, especially if you have a lot of articles so be aware of that.
I’m not going to go through all the parameters for the NMF model I’m using here, but they do impact the overall score for each topic so again, find good parameters that work for your dataset. You could also grid search the different parameters but that will obviously be pretty computationally expensive.
After the model is run we can visually inspect the coherence score by topic
30 was the number of topics that returned the highest coherence score (.435) and it drops off pretty fast after that. Overall this is a decent score but I’m not too concerned with the actual value. The real test is going through the topics yourself to make sure they make sense for the articles.
10 topics was a close second in terms of coherence score (.432) so you can see that that could have also been selected with a different set of parameters. So, like I said, this isn’t a perfect solution as that’s a pretty wide range but it’s pretty obvious from the graph that topics between 10 to 40 will produce good results. That said, you may want to average the top 5 topic numbers, take the middle topic number in the top 5 etc. For now we’ll just go with 30.
Another challenge is summarizing the topics. The best solution here would to have a human go through the texts and manually create topics. This is obviously not ideal. Another option is to use the words in each topic that had the highest score for that topic and them map those back to the feature names. I’m using the top 8 words. Here’s what that looks like:
We can them map those topics back to the articles by index.
For some topics, the latent factors discovered will approximate the text well and for some topics they may not. We can calculate the residuals for each article and topic to tell how good the topic is.
The residuals are the differences between observed and predicted values of the data. A residual of 0 means the topic perfectly approximates the text of the article, so the lower the better.
To calculate the residual you can take the Frobenius norm of the tf-idf weights (A) minus the dot product of the coefficients of the topics (H) and the topics (W). We can then get the average residual for each topic to see which has the smallest residual on average.
# Get the residuals for each documentr = np.zeros(A.shape[0])for row in range(A.shape[0]):    r[row] = np.linalg.norm(A[row, :] - H[row, :].dot(W), 'fro')# Add the residuals to the dfdf_topics['resid'] = r
Topic #9 has the lowest residual and therefore means the topic approximates the text the the best while topic #18 has the highest residual.
The summary for topic #9 is “instacart worker shopper custom order gig compani” and there are 5 articles that belong to that topic.
Links to the articles:
Workers say gig companies doing ‘bare minimum’ during coronavirus outbreak
Instacart makes more changes ahead of planned worker strike
Instacart shoppers plan strike over treatment during pandemic
Here’s why Amazon and Instacart workers are striking at a time when you need them most
Instacart plans to hire 300,000 more workers as demand surges for grocery deliveries
This is a very coherent topic with all the articles being about instacart and gig workers. The summary we created automatically also does a pretty good job of explaining the topic itself.
Now let’s take a look at the worst topic (#18). The summary is “egg sell retail price easter product shoe market”. There are 16 articles in total in this topic so we’ll just focus on the top 5 in terms of highest residuals.
Links to articles:
Crocs donating its shoes to healthcare workers
Want to buy gold coins or bars? Good luck finding any
Rothy’s has new idea for ocean plastic waste: handbags
Do you really need new clothes every month? Subscription box novelty has worn off
Americans are panic buying food for their pets
As you can see the articles are kind of all over the place. In general they are mostly about retail products and shopping (except the article about gold) and the crocs article is about shoes but none of the articles have anything to do with easter or eggs. They are still connected although pretty loosely.
Once you fit the model, you can pass it a new article and have it predict the topic. You just need to transform the new texts through the tf-idf and NMF models that were previously fitted on the original articles. Notice I’m just calling transform here and not fit or fit transform.
# Transform the new data with the fitted modelstfidf_new = tfidf_vectorizer.transform(new_texts)X_new = nmf.transform(tfidf_new)# Get the top predicted topicpredicted_topics = [np.argsort(each)[::-1][0] for each in X_new]# Add to the dfdf_new['pred_topic_num'] = predicted_topics
I continued scraping articles after I collected the initial set and randomly selected 5 articles. So these were never previously seen by the model. Overall it did a good job of predicting the topics.
US clears the way for this self-driving vehicle with no steering wheel or pedals - topic: solar comput technolog spacex energi power ibm vehicl
How to manage a team remotely during this crisis - topic: work home offic kid desk peopl school
Congress extended unemployment assistance to gig workers. But they’re struggling to access it - topic: uber worker employ govern self freelanc driver
Stelter: Federal response to pandemic is a 9/11-level failure - topic: trump fox news brief hanniti white hous presid
Nintendo pauses Nintendo Switch shipments to Japan amid global shortage - topic: game xbox video play graphic nbc olymp doom