Introduction to NLP - Part 3: TF-IDF explained | by Zolzaya Luvsandorj | Towards Data Science

Term frequency-inverse document frequency, also known as tf-idf... 💤
Does this sound gibberish to you? But you wish it doesn’t? In this post, I will first demonstrate how to vectorise text data to tf-idf using sklearn and then show a step-by-step process on how do it ourselves without any software. I hope tf-idf will be clearer to you by the end of this post! 🎓
First things first, let’s familiarise with some definitions to ensure that we are aligned on what each concept means:
* tf and tf-idf were broken into two variations: one based on count (_raw) and the other on percentage to make things clear.
** A more generic definition for idf would be a weight that upweights less frequent terms and downweights more frequent terms. However, I have chosen a simple definition because the lowest weight a term can have is 1 based on the formulas being used. Either way, less frequent terms are upweighted and more frequent terms are downweighted in tf-idf compared to tf.
The examples in brackets show references to the dataset used in this post.
To keep things manageable, we will create 2 small documents from these sentences which will allow us to monitor inputs and outputs for each step:
d1 = 'I thought, I thought of thinking of thanking you for the gift'd2 = 'She was thinking of going to go and get you a GIFT!'
If you are wondering why I have chosen these two sentences, I had to think of the smallest example that translated into my ideal end vector after they are preproccesed and think backwards. A tongue twister came in handy!
This section assumes that you have access to and are familiar with Python including installing packages, defining functions and other basic tasks. If you are new to Python, this is a good place to get started.
I have used and tested the scripts in Python 3.7.1. Let’s make sure you have the right tools before using the code.
We will use the following powerful third party packages:
pandas: Data analysis library,
nltk: Natural Language Tool Kit library and
sklearn: Machine Learning library.
The script below can help you download these corpora. If you have already downloaded, running this will notify you that it is up-to-date:
import nltknltk.download('stopwords')nltk.download('wordnet')
Firstly, let’s prepare the environment with packages and data:
# Import packages and modulesimport pandas as pdfrom nltk.stem import WordNetLemmatizerfrom nltk.tokenize import RegexpTokenizerfrom nltk.corpus import stopwordsfrom sklearn.feature_extraction.text import TfidfVectorizer# Create a dataframeX_train = pd.DataFrame({'text': [d1, d2]})
From now on, we will refer ‘X_train’ as our corpus (despite its size) and two sentences as documents. Second, we will need a text processing function to pass it on to TfidfVectorizer:
def preprocess_text(text):    # Tokenise words while ignoring punctuation    tokeniser = RegexpTokenizer(r'\w+')    tokens = tokeniser.tokenize(text)        # Lowercase and lemmatise     lemmatiser = WordNetLemmatizer()    lemmas = [lemmatiser.lemmatize(token.lower(), pos='v') for token in tokens]        # Remove stopwords    keywords= [lemma for lemma in lemmas if lemma not in stopwords.words('english')]    return keywords
🔗 I have explained this function in detail in the first part of the series if you need explanation. This preprocessor will transform the documents to:
d1 = [‘think’, ‘think’, ‘think’, ‘thank’, ‘gift’]d2 = [‘think’, ‘go’, ‘go’, ‘get’, ‘gift’]
Lastly, preprocess the corpus:
# Create an instance of TfidfVectorizervectoriser = TfidfVectorizer(analyzer=preprocess_text)# Fit to the data and transform to feature matrixX_train = vectoriser.fit_transform(X_train['text'])# Convert sparse matrix to dataframeX_train = pd.DataFrame.sparse.from_spmatrix(X_train)# Save mapping on which index refers to which wordscol_map = {v:k for k, v in vectoriser.vocabulary_.items()}# Rename each column using the mappingfor col in X_train.columns:    X_train.rename(columns={col: col_map[col]}, inplace=True)X_train
Once the script is run, you will get this output:
Tada❕ We have vectorised the corpus to tf-idf! The data is now in a format that is acceptable for a machine learning model.
Let’s see how test documents with or without unseen terms get transformed:
d3 = “He thinks he will go!”d4 = “They don’t know what to buy!”# Create dataframeX_test = pd.DataFrame({‘text’: [d3, d4]})# Transform to feature matrixX_test = vectoriser.transform(X_test['text'])# Convert sparse matrix to dataframeX_test = pd.DataFrame.sparse.from_spmatrix(X_test)# Add column names to make it more readiblefor col in X_test.columns:    X_test.rename(columns={col: col_map[col]}, inplace=True)X_test
When preprocess_text is applied, the test documents will transform into:
d3 = [‘think’, 'go'] # vectoritiser is familiar with these termsd4 = [‘know’, ‘buy’] # vectoritiser is not familiar with these terms
Is this what you expected to see when X_test is transformed? Although we have one occurrence of ‘go’ and ‘think’ in d3, do you notice how ‘go’ have been weighted up relative to ‘think’? Does the fact that d4 takes all 0 values make sense to you? Do you notice the number of terms in the matrix is dependent on training data much like any other sklearn transformers in general?
I think things get more interesting when we start looking under the hood. In this section, we will do the transformation manually, isn’t that fun? 😍
If you enjoy math, I encourage you to manually calculate problems alongside this guide, or better yet, try to calculate it on your own before proceeding to the answers that follow. In the examples below, rows represent either document or corpus and columns represent terms.
🔒 Problem: Count raw term frequency for each term by document.
💭 Hint: Look at preprocessed d1 and d2 and definition of tf_raw.
🔑 Answer:
This table summarises the number of times each term occurs in a document. For instance: We see 3 occurrences of ‘think’ in document 1 but only once in document 2. The number of columns are determined by the number of unique terms in the corpus. This step is actually what sklearn’s CountVectoriser does.
(This step is only to compare the final output against.)
🔒 Problem: Calculate term frequency for each term by document.
💭 Hint: What proportion does a term represent in a document? (Row %)
🔑 Answer:
The term ‘think’ takes 60% of the terms in document 1.
🔒 Problem: Count document frequency for each term.
💭 Hint: How many documents contain the particular term?
🔑 Answer:
The term ‘get’ is present in document 1 only while the term ‘think’ is in both documents.
Now, this is where the first calculation happens. ➗ Formula: In this formula, n stands for number of documents.
🔒 Problem: Calculate idf for each term.
🔑 Answer: Doesn’t idf look somewhat inverse to the df?
In sklearn, you can access idf_ attribute from the fitted TfidfVectorizer.
🔍 Example calculation: Below, I have provided an example calculation. You can use it as a guide for the remaining terms by replicating the same logic:
➗ Formula:
🔒 Problem: Calculate raw tf-idf (i.e. weighted counts) for each term by document.
🔑 Answer:
🔍 Example calculation:
➗ Formula:
🔒 Problem: Calculate tf-idf for each term by document.
🔑 Answer:
🔍 Example calculation:
Yay❕ We have obtained the exact same results! Of note, if you had rounded off anywhere in the interim steps, your final answer may not match precisely with that provided above due to rounding error.
📌 Exercise: See if you can calculate tf-idf for d3 and d4, and match it to the output from sklearn in previous section.
💭 Hint: (1) Count tf_raw - terms refer to the terms from training data, (2) Calculate tf-idf_raw using the idf we have built, (3) Calculate tf-idf. Do these steps only for the terms from training.
This method replicates output when smooth_idf=True for TfidfVectorizer or TfidfTransformer in sklearn. If you change this parameter to False, you will have to adjust the idf formula slightly by taking out +1 from both numerator and denominator.
Before we wrap up, let’s compare tf vs tf-idf for document 1:
Since ‘gift’ and ‘think’ are present in all documents, their relative weight is identical in both tables. However, ‘thank’ is only in document 1, so its relative frequency is higher in tf-idf when compared to ‘gift’ or ‘think’. As such, this demonstrates how tf-idf upweights terms that are in fewer documents and downweights those that are in more.
📌 Exercise: Analyse document 2 on your own.
Obviously, manual calculation is more prone to error and not likely to scale well to real corpus with hundreds, thousands or even millions of documents. Many thanks to sklearn contributors for providing such an efficient way to transform text to tf-idf in a just few lines of code. The manual calculations shown here was only to exemplify the underlying implementation when using a software. If you want to replicate outputs yourself from tools other than sklearn, an adjustment to the formulas may be necessary, but the overall idea should be similar. I hope these explanations have been useful and insightful.
Would you like to access more content like this? Medium members get unlimited access to any articles on Medium. If you become a member using my referral link, a portion of your membership fee will directly go to support me.
Thank you for taking the time to go through this post. I hope that you learned something from reading it. Links to the rest of the posts are collated below:◼️ Part 1: Preprocessing text in Python◼️ Part 2: Difference between lemmatisation and stemming◼️ Part 3: TF-IDF explained◼️ Part 4: Supervised text classification model in Python◼️ Part 5A: Unsupervised topic model in Python (sklearn)◼️ Part 5B: Unsupervised topic model in Python (gensim)
Happy transforming! Bye for now 🏃💨
Bird, Steven, Edward Loper and Ewan Klein, Natural Language Processing with Python. O’Reilly Media Inc, 2009
Feature Extraction, sklearn documentation