Bag Of Visual Words. Bag Of Visual Words(also known as Bag…
By: Vinayak Mishra
Tags: Computer Vision, Deep Learning, Bag Of Visual Words, Bag Of Words, AI

Bag Of Visual Words(also known as Bag Of Features) is a technique to compactly describe images and compute similarities between images.

It is used for image classification. The approach has its origin in text retrieval(information retrieval) and is an extension to the NLP algorithm Bag of Words. In Bag Of Words, we scan through the entire document and keep a count of each word appearing in the document. Then, we create a histogram of frequencies of words and use this histogram to describe the text document. In Bag Of Visual Words, our input is an image instead of text document and we use visual words to describe an image.

Bag Of Words for text documents

Bag Of Visual Words for Images

Visual words

In BoVW, we breakdown our image into a set of independent features.Features consists of keypoints and descriptors.Keypoints are the same thing as interest points. They are spatial locations, or points in the image that define what is interesting or what stand out in the image. They are invariant to image rotation, shrinkage, translation, distortion, and so on.Keypoints are specific points in an image say every 10th pixel in an image.Descriptors are the values(description) of that keypoints.We then create a dictionary or codebook based on these descriptors using clustering algorithms.We reiterate through our images and check whether our image has words present in the dictionary.If yes, we increase the count of that particular word.Finally we create the histogram for this image.From this histogram,we can find similar images or predict the category of this image.

How we create the codebook???…..

To create the codebook,we use feature extractors like SIFT,BRISK,etc.As discussed earlier, these techniques detect the keypoints in our image and compute their values(descriptors) for the input image.These feature detectors return an array containing the descriptors.We do this for every image in our training dataset.

Now, we will be having N(no of images in training dataset) arrays.In the next step, we stack these arrays vertically. Now we will use clustering algorithm like K-Means to form K clusters.K-Means groups the data points into K groups and will return the center of each group(see image below).Each cluster center(centroid) acts as a visual word.All these K centroids form our codebook.

K-Means Clustering

Creating Histograms……..

Detecting the visual words

Now we will create a 2D-array of zeros of shape (N,K) .We will see how to fill this array in the next few lines.Next we iterate through our images again and look for words in the image present in our dictionary.Once we detect a word present in both the dictionary and image, we increase the count of that particular word(i.e array[i][w]+=1 where i is the current image and w is the word).This is how we create the histograms for the images.

Creating the Histogram

Similarily,all the images will be converted to histograms.Now we can train a model(Random Forest,LinearSVC) for classifying the images.The images in the test dataset are preprocessed in the same manner and then fed to the model to predict the image category.

BoVW approach works well with large microscope images that capture many details

There is, however, a problem with this approach. It occurs when the visual words occur in a lot or every image of the image database. Consider a text document,words like is,are,etc won’t help much as they are mostly present in all the documents.These words make the classifying task harder. To solve this problem, we can apply the so-called TF-IDF (Term Frequency — Inverse Document Frequency) reweighting approach. It reweights every bin of a histogram and will downweight the “uninformative” words (i.e., features that occur in a lot of images/everywhere) and enhance the importance of rare words.

Every reweighted word in the histogram can be computed using the TF-IDF formula given in the below image.

TF-IDF weighting

If you play around with the formula for some time,you will understand how it gives less importance to words appearing in all the images.

In the above image,you can see how the blue word gets almost zero weight in the reweighted histograms.

So friends this is all I had to say……..

Please help me by providing your valuable feedback.