NMF — A visual explainer and Python Implementation | by Anupama Garla | Towards Data Science

Gain an intuition for the unsupervised learning algorithm that allows data scientists to extract topics from texts, photos, and more, and build those handy recommendation systems. NMF explanation is followed by a Python Implementation on a toy example of topic modelling on Presidential Inauguration Speeches.
“Is perception of the whole based on perception of its parts?”
Researchers Lee and Seung go on to lay out the mathematical basis of NMF in their 1999 paper in Nature — “Here we demonstrate an algorithm for non-negative matrix factorization that is able to learn parts of faces and semantic features of text.”
The beauty of data science is its ability to translate rather philosophical theory into mathematical algorithms that test the usefulness of such theory. NMF answers the question — to what extent a whole is a sum of its parts? And how do we use these parts? Companies typically use these parts to find patterns and associations between more complex whole objects.
In a business context, topics can be used to segment customers into types and target them differently. Additionally, the document-topic matrix can be used to plot each document in space and provide a system to give recommendations based on similarities and differences.
Let’s start looking at NMF in the context of natural language processing where it is often used for topic modeling.
We convert a corpus of documents into numbers to answer questions like:
What are the prevailing topics in this corpus?
To what extent is each topic important in each document?
How well do these extracted topics describe the original document?
How similar is one document to another?
There are many more questions that can be answered with NMF and natural language processing, but these are the core questions. First, let’s understand the terminology data scientists and a lot of our literature like to use.
We use corpus to refer to a group of similar media. For NMF, this could be a group of texts like articles, images like aerial photos or portraits, audio like songs, or youtube videos — anything that can be conceptualized as the addition of parts (rather than the addition and subtraction of parts). We use document to refer to one text, photo, song, or video. A term is a word.
NMF is a form of Topic Modelling — the art of extracting meaningful themes that recur through a corpus of documents. A corpus is composed of a set of topics embedded in its documents. A document is composed of a hierarchy of topics. A topic is composed of a hierarchy of terms.
NMF uses the logic of the ‘the distributional hypothesis’ discussed in detail in the Swedish linguist, Magnus Sahlgren’s paper, The Distributional Hypothesis.
Sahlgren offers one often cited explanation,
‘This hypothesis is often stated in terms like “words which are similar in meaning occur in similar contexts” (Rubenstein & Goodenough, 1965)’
and continues — ‘The general idea behind the distributional hypothesis seems clear enough: there is a correlation between distributional similarity and meaning similarity, which allows us to utilize the former in order to estimate the latter.’
For us, this means that documents can be treated as a bag of words. Similar documents have similar word frequency distributions — and similar topics. Two words that occur together are likely similar — belonging to a single topic. This algorithm ignores syntactic information aka word order, and semantic information aka multiple meanings.
Let’s dig in!
NMF stands for Latent Semantic Analysis with the ‘Non-negative Matrix-Factorization’ method used to decompose the document-term matrix into two smaller matrices — the document-topic matrix (U) and the topic-term matrix (W) — each populated with unnormalized probabilities.
The values in V can be approximated through matrix multiplication of W and H. For instance the grey box in V can be found through matrix multiplication of the first row of W by the first column of H, also greyed in. The first row of V can be approximated through matrix multiplication of the first row of W with the entire matrix of H.
V is for Visible
In the V matrix, each row represents a Document that is composed of a frequency of Terms aka a count of each word’s occurence. Each column represents the visible variables. For image processing, the visible variables would be specific pixels. Here is an excerpt of a V Matrix of Presidential Inauguration Speech Text:
W is for weights
In the W matrix, each row represents a Document that is composed of unnormalized probabilities of Topics. Each column represents a semantic feature that recurs throughout the corpus. For image processing, the features would be specific prototypical characteristics like ‘mustache’. Here is an excerpt of a W Matrix of Presidential Inauguration Speech Text:
H is for hidden
In the H matrix, each row represents a Topic or semantic feature that is composed of term frequencies. Each column represents a visible variable. Two terms that occur frequently together form a topic, and each term gives more contextual meaning for the term it is grouped together with. If a term occurs frequently in two topics, then those topics are likely related. Here is an excerpt of an H Matrix of Presidential Inauguration Speech Text:
Reconstructing V
The W and H matrix can be used to approximately reconstruct V through matrix multiplication.
The output of NMF changes each time you run it, and the topics are not resolved — the data scientist must infer the topic from the highest word frequencies per topic, using the H matrix. This is where the art of choosing the correct number of topics comes into play. Too many, and you have topic repeats or topics composed of sparsely related words. Too few, and you have not very meaningful topics.
In the example above, the Topics (p) are set to 6. Each column of the W matrix represents a probability that the topic is in the document. Each row of the W matrix represents a distribution of topic frequencies in each Document. Each row of the H matrix represent the distribution of term frequencies in each topic, and can be seen as the degree to which each term is activated in each topic.
NMF Math
Like most machine learning algorithms, NMF operates by starting with a guess of values for W and H, and iteratively minimizing the loss function. Typically it is implemented by updating one matrix (either W or H) for each iteration, and continuing to minimize the error function, ||V — WH || = 0, while W and H values remain non-negative, until W and H are stable. There are different loss functions used to update the matrices based on the specific programming package you use, but the one proposed in the original Lee and Seung paper is the multiplicative update rule. Honestly the math is fairly complex, and it would take me some study to wrap my head around it and explain it in a no-fuss style. I’m going to call it out-of-scope of this article. :-).
NMF caveats
NMF requires that all documents are fairly similar — for instance if you are comparing faces, you would look at faces from a similar angle. If you are comparing texts, you would look at texts of similar lengths. For more complex documents/images, you would need more layers of hidden variables, and that project would venture into the arena of neural networks.
NMF and TF-IDF
The advantage of NMF, as opposed to TF-IDF is that NMF breaks down the V matrix into two smaller matrices, W and H. The data scientist can set the number of Topics (p) to determine how small these matrices get. Data scientists often use the TF-IDF derived Document-Term Matrix as the Input Matrix, V, because it yields better results.
NMF vs. other matrix decomposition methods
NMF differs from other matrix decomposition methods like PCA and VQ in that it only uses non-negative numbers. This allows for each Topic or feature to be interpretable. Additionally, W and H can be represented by sparse matrices where only the values > 0 are encoded, making for a smaller dataset.
This is generally the order of operations:
Import Relevant Libraries to Jupyter Notebook / Install Libraries in your EnvironmentSelect Data and ImportIsolate Data to Topic ModelClean DataCreate Function to Pre-process Data*Create Document Term Matrix ‘V’Create Function to Display Topics*Run NMF on Document Term Matrix ‘V’Iterate until you find useful Topics
Import Relevant Libraries to Jupyter Notebook / Install Libraries in your Environment
Select Data and Import
Isolate Data to Topic Model
Clean Data
Create Function to Pre-process Data*
Create Document Term Matrix ‘V’
Create Function to Display Topics*
Run NMF on Document Term Matrix ‘V’
Iterate until you find useful Topics
*Star indicates optional steps
For dataframe manipulation and exporting you will need:
pandas
import pandas as pd
From scikit learn for modeling you will need:
TfidfVectorizer
NMF
text
from sklearn.feature_extraction.text import TfidfVectorizerfrom sklearn.decomposition import NMFfrom sklearn.feature_extraction import text
From nltk for text processing you may need:
stopwords
word_tokenize
pos_tag
from nltk.corpus import stopwordsfrom nltk import word_tokenize, pos_tagfrom nltk.stem import WordNetLemmatizer
For text cleaning you may need:
re (regular expressions)
string
import reimport string
Next, select your corpus of documents. I am using American President’s Inauguration Speeches, available as a Kaggle Dataset. I did get an error when I tried to import the speeches without the engine clause but I just googled my error and took this ‘engine’ suggestion which worked for me.
# expand pandas df column display width to enable easy inspectionpd.set_option('max_colwidth', 150)# read in csv to dataframedf = pd.read_csv('inaug_speeches.csv', engine='python')# visually inspect dataframedf.head()
I’m going to make a dataframe of the President’s names and speeches, and isolate to the first term inauguration speech, because some President’s did not have a second term. This makes for a corpus of documents with similar lengths.
# Select Rows that are first term inaugural addressesdf = df.drop_duplicates(subset=['Name'], keep='first')# Clean Up Indexdf = df.reset_index()# Select only President's Names and their Speechesdf = df[['Name', 'text']]# Set Index to President's Namesdf = df.set_index('Name')# Visually Inspectdf.head()
I want to make all of the text in the speeches as comparable as possible so I will create a cleaning function that removes punctuation, capitalization, numbers, and strange characters. I use regular expressions for this, which offers a lot of ways to ‘substitute’ text. There are tons of regular expression cheat sheets, like this one. I ‘apply’ this function to my speech column.
def clean_text_round1(text):    '''Make text lowercase, remove text in square brackets,     remove punctuation, remove read errors,    and remove words containing numbers.'''    text = text.lower()    text = re.sub('\[.*?\]', ' ', text)    text = re.sub('[%s]' % re.escape(string.punctuation), ' ', text)    text = re.sub('\w*\d\w*', ' ', text)    text = re.sub('�', ' ', text)    return textround1 = lambda x: clean_text_round1(x)# Clean Speech Textdf["text"] = df["text"].apply(round1)# Visually Inspectdf.head()
Here we will lemmatize all the words in the speeches so that different forms of a particular word will be reduced to a base form, for instance noun plurals become singular and all verb tenses become present. This simplifies the text so that repeated instances of slight variations of a word are interpreted as one word. One can stem or lemmatize, more info here. The other thing we are doing here is isolating the speech text to a particular part of speech — nouns. You can experiment with different parts of speech isolation but I have found most success with nouns in this particular dataset. More info on how to select parts of speech with ‘pos’ (part of speech) tagging here. This is a step you can skip if you want to setup a basic pipeline first and then you can add it in when you iterate to find what works best with your dataset.
# Noun extract and lemmatize functiondef nouns(text):    '''Given a string of text, tokenize the text     and pull out only the nouns.'''    # create mask to isolate words that are nouns    is_noun = lambda pos: pos[:2] == 'NN'    # store function to split string of words     # into a list of words (tokens)    tokenized = word_tokenize(text)    # store function to lemmatize each word    wordnet_lemmatizer = WordNetLemmatizer()    # use list comprehension to lemmatize all words     # and create a list of all nouns    all_nouns = [wordnet_lemmatizer.lemmatize(word) \    for (word, pos) in pos_tag(tokenized) if is_noun(pos)]         #return string of joined list of nouns    return ' '.join(all_nouns)# Create dataframe of only nouns from speechesdata_nouns = pd.DataFrame(df.text.apply(nouns))# Visually Inspectdata_nouns.head()
Here I added some stopwords to the stopword list so we don’t get words like ‘America’ in the topics as that is not super meaningful in this context. I also use TF-IDF Vectorizer rather than a simple Count Vectorizer in order to give greater value to more unique terms. You can learn more about TF-IDF here.
# Add additional stop words since we are recreating the document-term matrixstop_noun = ["america", 'today', 'thing']stop_words_noun_agg = text.ENGLISH_STOP_WORDS.union(stop_noun)# Create a document-term matrix with only nouns# Store TF-IDF Vectorizertv_noun = TfidfVectorizer(stop_words=stop_words_noun_agg, ngram_range = (1,1), max_df = .8, min_df = .01)# Fit and Transform speech noun text to a TF-IDF Doc-Term Matrixdata_tv_noun = tv_noun.fit_transform(data_nouns.text)# Create data-frame of Doc-Term Matrix with nouns as column namesdata_dtm_noun = pd.DataFrame(data_tv_noun.toarray(), columns=tv_noun.get_feature_names())# Set President's Names as Indexdata_dtm_noun.index = df.index# Visually inspect Document Term Matrixdata_dtm_noun.head()
To evaluate how useful the topics created by NMF are, we need to know what they are. Here I create a function to display the top words activated for each topic.
def display_topics(model, feature_names, num_top_words,\ topic_names=None):'''Given an NMF model, feature_names, and number of top words, print topic number and its top feature names, up to specified number of top words.'''    # iterate through topics in topic-term matrix, 'H' aka    # model.components_    for ix, topic in enumerate(model.components_):        #print topic, topic number, and top words        if not topic_names or not topic_names[ix]:            print("\nTopic ", ix)        else:            print("\nTopic: '",topic_names[ix],"'")        print(", ".join([feature_names[i] \             for i in topic.argsort()[:-num_top_words - 1:-1]]))
Perhaps the easiest step is running NMF on the document term matrix once you have one.
nmf_model = NMF(2)# Learn an NMF model for given Document Term Matrix 'V' # Extract the document-topic matrix 'W'doc_topic = nmf_model.fit_transform(data_dtm_noun)# Extract top words from the topic-term matrix 'H' display_topics(nmf_model, tv_noun.get_feature_names(), 5)
This is where the art comes in. As a basic dummy model, I usually run the text through a count vectorizer and then NMF to get an idea of what we are looking at. We can iterate through topic numbers, parts of speech, TF-IDF settings, adding stop words and likely so much more. Getting useful topics can be done by eye, and is definitely a challenge.
nmf_model = NMF(8)doc_topic = nmf_model.fit_transform(data_dtm_noun)display_topics(nmf_model, tv_noun.get_feature_names(), 5)
Creating useful topics is challenging, even with the help of algorithms.
However, here we can see 8 inauguration themes which could be described as — laws, peace, leadership, spending, justice, revenue, ideals, decision-making. This is just the beginning of a project that would investigate inauguration speeches. Some questions we can ask of these results are:
Which President focussed on which topics?
Which President’s speeches are most similar?
Is there a clear divide between recent Democratic and Republican Presidents in terms of topic focus? Is there a clear progression of topics through time?
The opportunities are many and the process is iterative, depending on what questions you seek to answer. The surprising results of this analysis so far is that recent Presidents, regardless of political party, discuss the same general subjects in relation to all Presidents in the past. That’s both surprising and expected as we are all living through similar challenges at this moment. I wonder what topic modeling on adjectives would yield...
On TF-IDF
towardsdatascience.com
On Creating a Recommendation System from NMF generated Topics
towardsdatascience.com
NMF vs. other topic modeling methods
There are several prevailing ways to convert a corpus of texts into topics — LDA, SVD, and NMF. You can read this paper explaining and comparing topic modeling algorithms to learn more about the different topic-modeling algorithms and evaluating their performance. I have experimented with all three methods, and found that NMF run on a TF-IDF matrix generated the most interpretable results in the least amount of run-time, which is why I dove into NMF in this article.
LDA : Latent Dirichlet Algorithm — probability based and decomposes the corpus into two matrices. This is a generative model where you can use these smaller matrices to reconstruct the original text. This requires a very large dataset and more run-time to get useful results in my experience.
SVD: Latent Semantic Analysis with Singular Value Decomposition — a way of decomposing the document-term matrix into three smaller matrices that can be reformatted into two matrices containing both negative and positive unnormalized probabilities which prevents a direct interpretation of the decomposed matrices.
NMF : Latent Semantic Analysis with Non-Negative Matrix Factorization — a way of decomposing the document-term matrix into two smaller matrices that contain only positive values which allows direct interpretation of each matrix as unnormalized probabilities.
NMF Original Paper
Even though there are tons of resources online for learning NMF, I think it’s always best to go to the source for a deeper understanding, so below is a link to the original Lee and Seung paper on NMF:
Lee, D., Seung, H. Learning the parts of objects by non-negative matrix factorization. Nature 401, 788–791 (1999). https://doi.org/10.1038/44565
NMF scikit learn Documentation
It’s also best to get acquainted with the toggles on your NMF algorithm in scikit learn. Dig in here once you start iterating.
Topic Supervised NMF
This method is a supervised spin on NMF and allows more user control over the topics. I haven’t dug into it yet, but would love to know if anyone has!