Getting Started in Natural Language Processing: Bag-of-Words & TF-IDF
By: Rax Automation Suite
Tags: Bag Of Words, Tf Idf, Machine Learning, Natural Language Process, Data Science

by Russ delos Santos

Several applications of machine learning, like image and audio processing, have already been analyzing vectorized data right from the get-go. These specific data have values (magnitude and direction) that can already provide meaningful information to the algorithm, like the noise level of the audio signal or the value and position of the pixel in the image, thereby enabling the easier computation.

However, it’s a different case for texts, particularly the human language. Machine learning models cannot work directly with raw texts! This problem now posits another dilemma — how can we transform texts into meaningful numbers or vectors such that they will be useful to ML pipelines? Simply transforming raw texts into, for example, binary, decimal, or hexadecimal representations, definitely won’t give us functional representations of words, since those values cannot capture information about the meaning, context, correlation between words such as ‘king’ and ‘queen’ or ‘cat’ and ‘dog’. Applying a model that represents a set of documents as vectors in a common vector space can solve this problem.

Vector Space Model is a multidimensional representation of words in a continuous vector space. The words closely or semantically related to each other tend to be nearer to each other in the model.

Figure 2. VSM sample visualization. Source: Polyvyanny & Kuropka, 2009

There are two models that can be used to “convert” words into vectors:

Count-based model — a computation model that relies on the frequency of words within a document. Predictive model — a model that aims to predict the context or the target word based on the neighboring terms.

Let us first tackle vectorization using count-based models called the bag-of-words (BoW) and the term frequency-inverse document frequency (TF-IDF).

Bag-of-Words Model

The bag-of-words model (BoW) is a vectorization technique that uses the number of occurrences of words within a document or a corpus, which is essentially defined as the dataset of known words. Take the set of documents below as examples.

“I am the king.”

“The kingdom is prosperous.”

“She is the queen.”

“The royal queen is beautiful.”

The corpus can now be made by listing all the unique words in this set of documents. Commonly, the first word to be read is the first one to be listed in the corpus. It is important to take note that BoW does not care about the order of the words. Thus, our corpus will now be:

Finally, to transform a sample text into a vector using BoW, each occurrence of the corpus words in the sample text is counted, and is outputted as an array of integer counts with size same as the corpus. For example, the sample text “The king and the queen rule the kingdom” will have a vector representation of:

[ 0, 0, 3, 1, 1, 0, 0, 0, 0, 0, 0 ]

as illustrated in the word count below:

Term Frequency-Inverse Document Frequency

The TF-IDF, an advanced variant of BoW, takes into account the importance of the words based on its rarity in the document. However, instead of vectors with discrete counts like BoW, the converted vector now contain continuous in value. This idea is best explained through the following equations:

tf weight is a normalized counting of words. Normalization basically means that it gets the relative count of words to avoid bias towards longer documents. It is derived from the equation:

So, suppose we have a document with a length of 150 words and a target term ‘dog’. If ‘dog’ appeared in the document 5 times, the tf weight of ‘dog’ would be:

idf weight measures the significance of the word in a corpus by giving frequently appearing words low weights, and vice-versa for rarely appearing words. It uses a logarithmic function with a formula:

Going back to the example, if the term ‘dog’ appeared in 10,000 documents out of a 1 million-document dataset, the idf weight would be:

Finally, the tfidf weight of the term ‘dog’ is

.

Implementation

Framework Used: .NET 4.7.2, Accord.NET v3.8

To understand how the words are transformed into vectors using the concepts above, I tried to implement BoW + tfiDF using C#. Specifically, I used the Accord.NET Framework since it enables an easier implementation of the theory.

After diving into coding, I installed the necessary packages needed for the implementation. I imported pertinent namespaces by calling the ‘using’ directive. For the Accord.NET, particularly the Accord.MachineLearning namespace, I used the NuGet Package Manager. In general, the final namespaces included would look like this:

I created a simple document set as presented below.

According to the theory, I should have the bag of words of the whole corpus. Therefore, the words were ‘tokenized’ and saved it into a jagged string array named document_words using the Tokenize() Method. I did get the individual words from the corpus.

Then, I tried to learn the words above by making an instance of tfidf using new TFIDF(), as shown below. To initialize the math discussed earlier, I chose TermFrequency.Default and InverseDocumentFrequency.Default.

Next, I called the method Learn() and placed the document_words as its parameter to transform the corpus into tfidf values. This will also remove the repeating words since we only need one instance of every word in the vocabulary.

We finally have a vector that contains 11 values:

[ tfidf_1, tfidf_2, tfidf_3, ……. , tfidf_11 ]

After the process, I tested it with texts (shown below) that were different from the dataset we had just learned.

Transforming new texts into tfidf based from previously learned dataset is fairly easy using Accord.NET. First, I tokenized words similar to the process earlier, and used it as the parameter forcalling the Transform() method.

Printing the values of test_weights would give us the vector representation of the test sentences.

Exploring how to transform words into vectors can be a good entry point to natural language processing, as it did for me with this project. Choosing a fairly easier mathematical transformation like BoW+tfidf can give you an insight into how words correlate with each other in the text, and in the corpus itself. You can leverage more on this step by finally feeding the data into a machine learning pipeline for classification or prediction tasks.

References:

A. Polyvyanyy and D. Kurokpka, “A Quantitative Evaluation of the Enhanced Topic-Based Vector Space Model.”