“Sklearn’s TF-IDF” vs “Standard TF-IDF”
By: Sivakar Siva
Tags: Sklearn, NLP, Math, Data Science, Machine Learning

This post was inspired back when I was trying to compute the TF-IDF matrix by hand using the standard formula, after computing TF and IDF first and then multiplying the two I did realize, there was something different compared to the one I obtained with Scikit-learn on my sample corpus, there I realized the difference between the Scikit-learn version and most standard and traditional version. Without further ado, let’s understand the differences.

What is the standard definition of TF-IDF?

TF-IDF definition: “term frequency-inverse document frequency, is a numerical statistic that is intended to reflect how important a word is to a document in a collection or corpus.[2]

Objective: The goal of using TF-IDF instead of the raw frequencies of occurrence of a token in a given document is to scale down the impact of tokens that occur very frequently in a given corpus and that are hence empirically less informative than features that occur in a small fraction of the training corpus.[1]

How to Compute to it?

It is simply a product of two weights, the TF and the IDF weight:

TF: Term Frequency measures how frequently a term occurs in a document as every document is different in length, it is possible that a term would appear much more times in long documents than shorter ones. Thus TF is:

IDF: Inverse Document Frequency measures how important a term is. In TF, all terms are considered equally important. Thus we need to weigh down the frequent terms while scaling up the rare ones, by computing the following:

Note: In Scikit-learn ,The log is not base 10 , though it is the natural logarithm (which has a base e, e is an irrational and transcendental number approximately equal to 2.718),

Here, there is a small example of what IDF means practically: The example has 4 terms: (a, boat, mobile and mobilegeddon).

Source: Moz by Eric Enge

You can see that, the IDF is highest with the term that is the rarest(Mobilegeddon), whereas less the frequency of the term smaller the IDF value for that term will be.

Scikit-learn TF-IDF

Now, How the TF-IDF computed in Scikit-learn’s TfidfTransformer and TfidfVectorizer differ slightly from the standard textbook notation?

Here are the differences: TF remains the same, while IDF is different:“ Some constant 1” is added to the numerator and denominator of the IDF as if an extra document was seen containing every term in the collection exactly once, which prevents zero divisions” a more empirical approach, while the standard notation present in most textbooks doesn’t have the constant 1.

Now, Let’s try applying Scikit learn TF-IDF step by step to the corpus formed by three sentences below and see the results:

In a text corpus, some words such as (e.g. “some”, “a”, “has”,etc… in English) if fed directly to a classifier, these very frequent terms would shadow the frequencies of rarer yet more interesting terms. Hence removing these using stopwords removal it is the best and effective approach since they carry very little information about the actual contents of the document.

The vocabulary forming all the unique words from the corpus would consist of these 5 words: “John”, “Cat”,” Eat”, “Fish” and “Big”.

Vocabulary Frequency Count

Term Frequency (TF)

Let’s start by calculating Tf(‘John’, sentence1)

Numerator: “John” occurs only once in sentence1

“John” occurs only once in sentence1 Denominator: Total Number of words in sentence1 are:{John, cat}=2

TF(‘John’, sentence1) = 1/2

Similarly, the TF values for the entire vocabulary would be the table below:

Resulting TF table

Inverse Document Frequency (IDF)

IDF, as stated above is a measure of how important a term is. IDF value is essential because computing just the TF alone is not enough to understand the importance of words.

Let’s calculate IDF(“John”) :

Numerator: Total Number of sentences =3

Total Number of sentences =3 Denominator: Number of sentences containing “John”=1

IDF(‘John’) = log((3+1)/(1+1))+1 = 1.69

Similarly, the IDF values for the entire vocabulary would be the table below:

Resulting IDF table

Applying (TF X IDF) Product

Normalization example calculated for the first vector-sentence

The final Tf-IDF vector table

Sklearn_TF-IDF output for the same corpus

Normalization Step

In Scikit-Learn, the resulting TF-IDF vectors are then normalized by the Euclidean norm. This was originally a term weighting scheme developed for information retrieval (as a ranking function for search engines results) that has also found good use in document classification and clustering.[1]

Summary

The main difference between the traditional and the Scikit-learn ‘s TF-IDF is the presence of the unitary constants in the denominator and numerator.

Standard vs Sklearn TF-idf result matrix

Overall this would not change the vector representation of the TF-IDF matrix, the vocabulary will be represented by higher weights compared to the standard one whereas in the Standard TF-IDF table the corresponding values are set to be smaller instead.

However, The core principle of TF-IDF would remain the same: TF-IDF gives larger values for less frequent words and is high when both IDF and TF values are high, as we can confirm for our less frequent words in our cor: {John and Big}: for both of them TF-IDF achieved a value bigger than the remaining higher frequency words in the TF-IDF table.

Overall until 2015, the rise word embeddings, TF-IDF has been the traditional approach to assigning weights to words that signify the relevance in documents compared to the traditional Bag of words approach that treats every word as equally important.t is still actively used in approaching NLP problems effectively. Below there is the code of the Sklearn implementation.

References

[1] https://scikit-learn.org/stable/modules/feature_extraction.html#text-feature-extraction

[2]https://en.wikipedia.org/wiki/Tf%E2%80%93idf

[3]Moz. 2020. Inverse Document Frequency And The Importance Of Uniqueness. [online] Available at: https://moz.com/blog/inverse-document-frequency-and-the-importance-of-uniqueness