TF-IDF : A visual explainer and Python Implementation on Presidential Inauguration Speeches | by Anupama Garla | Towards Data Science

Ever asked to explain TF-IDF to non-technical audiences? Here is a visual unpacking of TF-IDF (Term Frequency — Inverse Document Frequency) to share with non-technical colleagues and gain an intuition for the equation that drives ranking search engines from Google to Amazon, and many industry standard Natural Language Processing algorithms. Python example and implementation follows.
We convert a group of texts into numbers to answer questions like:
What are the prevailing themes in each of these documents?
Can you order the documents from most similar to least?
What are the most meaningful words being used?
There are many more questions that can be answered with TF-IDF and Natural Language Processing. But first, let’s understand the terminology data scientists and a lot of our literature like to use.
Terminology
We use document to refer to the pieces of text that we are comparing — these could be newspaper articles, chapters of a book, product descriptions, reviews, homepages, etc. Corpus is a group of such documents. A term is a word.
The two ways to convert words to meaningful numbers are the ‘bag of words’ count vectorizer approach, and TF-IDF vectorizer. Count vectorizer ranks words based on how many times each word appears in each document. TF-IDF takes into account how many words are in the document in relation to this count, as well as how frequently this word is in a document in the corpus. Therefore the TF-IDF value indicates the relative importance of each word in a corpus. Let’s look at how to derive this value.
IDF — Inverse Document Frequency
Is this term a common theme in all documents within our corpus?
IDF answers this question. A value close to 0 indicates the term is very common — and not very helpful in differentiating this document from another. A higher value indicates the term is not very common and could be crucial to understanding the theme of this document.
TF — Term Frequency
Is this term important in this document?
The TF answers this question. A value close to 1 indicates this term is very important to the document — and primarily composed of it. A low value could mean this term is not very important.
TF-IDF — Term Frequency-Inverse Document Frequency
What is the importance of this term in this document, which also accounts for the frequency of this term in the corpus?
TF-IDF answers this question. A value close to 0 indicates the term is not important either in the corpus or the document or both. A larger value indicates the term is unique for the document or the corpus or both. This value works best in relation to other terms in the same document, and other documents.
TF-IDF Matrix
How can I use TF-IDF values to generate insights about documents and terms?
Corpus TF-IDF Values become valuable when you are able to compare terms to each other across documents. You can tell which documents are similar, and which are different — which documents have nothing novel in them, and which ones do. Terms that are in fewer documents, have higher scores, whereas terms in all documents have a score of 0.
Term Document Matrix
By examining the Term Document Matrix that shows the TF-IDF Values for each term in each document, we can learn which documents are most similar, and which terms are most important. This is useful when searching for an item on Google or Amazon, or creating a recommendation system that finds the most similar books, movies, partners, etc.
The uses of the TF-IDF Document — Term matrix are endless, and is built on a relatively simple equation. The power of machine learning is taking these little logics and repeating them across huge datasets to generate new products, tools, insight, and impact. Heck, even small datasets can be used to generate insight. Here is an example in python that looks at Presidential Inauguration Speeches, which were extracted from The American Presidency Project at UCSB.
First, you want to import the Python Libraries you will need.
TfidfVectorizer from the scikit learn library to process text
pandas to use dataframes
from sklearn.feature_extraction.text import TfidfVectorizerimport pandas as pd
Next, select your corpus of documents. I extracted text from the previous 4 President’s inauguration addresses and stored each of them in strings. My corpus list is a list of those strings.
corpus = [document_bush_2001, document_obama_2009,   document_trump_2017, document_biden_2021]
There is a lot of options to pre-process your strings by using stemming or lemmatization, but I will skip that here as the focus is on TF-IDF. To use TF-IDF Vectorizer, you store the model in a variable, where you can set many settings. Some starter settings are stop words, ngrams, min and max df. I recommend looking into the scikit learn documentation to find out more.
Stop words are common words you don’t want to consider like ‘the’ or ‘of’. You can always add to stop words, but I find that min and max df takes care of this automatically.
N-grams allows you to choose a range of word lengths — 1, 2, or more — I’m sticking with single words
max-df ignores terms whose document frequency is greater than the selected threshold
min-df ignores terms whose document frequency is less than the selected threshold
vectorizer = TfidfVectorizer(stop_words='english', ngram_range = (1,1), max_df = .6, min_df = .01)
Apply TfidfVectorizer to speech texts. X is a sparse matrix which holds the location of all non-zero values.
X = vectorizer.fit_transform(corpus)
Extract words aka ‘feature names’ from vectorizer.
feature_names = vectorizer.get_feature_names()
Create a dense matrix to convert into a list, and finally a dataframe.
dense = X.todense()denselist = dense.tolist()df = pd.DataFrame(denselist, columns=feature_names)df.head()
My TF-IDF Document-Term Matrix output looks like this:
There it is! Not very interesting, right? What about if we name the Documents, order the words by highest TF-IDF score per Document, and show the 15 most meaningful words by Presidential Inauguration Speech? Let’s see:
data = df.transpose()data.columns = ['document_bush_2001', 'document_obama_2009', 'document_trump_2017', 'document_biden_2021']# Find the top 30 words said by each Presidenttop_dict = {}for c in range(4):    top = data.iloc[:,c].sort_values(ascending=False).head(30)    top_dict[data.columns[c]]= list(zip(top.index, top.values))# Print the top 15 words said by each Presidentfor president, top_words in top_dict.items():    print(president)    print(', '.join([word for word, count in top_words[0:14]]))    print('---')
For NLP lovers and political buffs, these results are very revealing. Words are powerful, and we can clearly see what the focus of each candidate was following their election, as well as the state of the union. For instance, Trump is focussed on wealth, dreams, and left vs right. Biden is focussed on democracy, soul, the virus and loss, and doing better. One interesting theme is ‘story’ in both Bush and Biden’s speeches. I wonder what each of them were saying about ‘story’ in their speeches. This is just the tip of the iceberg but I hope that you can see how powerful TF-IDF is at extracting meaning from text. In fact, I used a TFIDF Vectorizer to build Storytimes, a Content-Based Illustrated Children’s Book Recommender for Parents, which you can read about here.
For data scientists in general, the methodology that went into generating TF-IDF can be used in non-textual applications. For instance, if you want to do some feature engineering on already numerical data to make already meaningful features more powerful, you can. For instance let’s say you find that the number of bathrooms and bedrooms in a house are important features in predicting the price of the house, you can engineer an even more predictive feature than the raw number of bathrooms or bedrooms. you can look at the density of bathrooms per bedrooms, let’s call it bathroom_frequency = (2 bathrooms / 3 bedrooms) and multiply that by the rarity of this feature for the neighborhood, let’s call it inverse_neighborhood_frequency = log(# of neighborhood houses / # of neighborhood houses with 2 bathrooms) to get a ‘BF-INF’ feature. In fact, I did something similar in my feature engineering to predict the rental price of airbnb’s in LA, which you can read about here.
TF-IDF is great, and it’s applications are extensive inside and outside of Natural Language Processing.