Reverse Image Search — Find Similar images
By: Mani Kanta
Tags: Machine Learning, Python, Deep Learning, Resnet50, Image Processing

Photo by Format from Pexels

Have you ever wondered how google image search works or How amazon can retrieve products similar to the image that we upload in the app/site?

To achieve this task we will be using one simple method.

We are going to pick a pre-trained deep learning model, remove the top layers, and extract the convolutional features for the images in our dataset. Then we will use these feature vectors to find similar images by using sklearn’s nearest neighbor algorithm.

Let’s get started.

Import libraries

import requests

import os

import numpy as np

from numpy.linalg import norm

import joblib as pickle

from tqdm import tqdm

import os

import PIL

import time

import tensorflow as tf

from tensorflow.keras.preprocessing import image

from tensorflow.keras.preprocessing.image import ImageDataGenerator

import gc

from tensorflow.keras.applications.resnet50 import ResNet50, preprocess_input

#from tensorflow.keras.applications.MobileNet import MobileNetV2,preprocess_input

#from tensorflow.keras.applications.mobilenet import MobileNet,preprocess_input

import math

from sklearn.neighbors import NearestNeighbors

import matplotlib.pyplot as plt

import matplotlib.image as mpimg

from sklearn.decomposition import PCA

We will be using Caltech101 object dataset. A small description of the dataset — Pictures of objects belonging to 101 categories. About 40 to 800 images per category. Most categories have about 50 images. Collected in September 2003 by Fei-Fei Li, Marco Andreetto, and Marc ‘Aurelio Ranzato. The size of each image is roughly 300 x 200 pixels.

To know more about the dataset checkout http://www.vision.caltech.edu/Image_Datasets/Caltech101/

url = 'http://www.vision.caltech.edu/Image_Datasets/Caltech101/101_ObjectCategories.tar.gz'

r = requests.get(url)

filename = '101_ObjectCategories.tar.gz'

folderdername = filename.split('.')[0]

open(filename , 'wb').write(r.content) # reference https://stackoverflow.com/questions/30887979/i-want-to-create-a-script-for-unzip-tar-gz-file-via-python

import tarfile

if filename.endswith("tar.gz"):

tar = tarfile.open(filename, "r:gz")

tar.extractall()

tar.close()

elif filename.endswith("tar"):

tar = tarfile.open(filename, "r:")

tar.extractall()

tar.close()

In the dataset we have ‘Background google’ folder which contains random google images. So let’s remove it.

#!rm -rf '101_ObjectCategories/BACKGROUND_Google' #Linux

#os.removedirs('101_ObjectCategories/BACKGROUND_Google') #windows

Now we will use Resnet50 model which is pre-trained on the imagenet dataset (http://www.image-net.org/) to extract the features from the Caltech101 images dataset. In other words we will convert the images in the dataset to feature vectors using Resnet50.

Why resnet ?

We chose resnet because of the relatively small feature size. It converts images into 2048 convolutional features compared to 25088 features in vgg 19 or 51200 features in inception architectures. It will be easy for the nearest neighbor algorithm to find neighbors and also it will minimize the effects of Curse_of_dimensionality. https://en.wikipedia.org/wiki/Curse_of_dimensionality

Create Resnet50 model without top layers so we get convolutional features as output instead of the image class probability.

img_size =224



model = ResNet50(weights='imagenet', include_top=False,input_shape=(img_size, img_size, 3),pooling='max')

Here are creating a Keras image data generator object and preprocessing the images. After that we are passing the data generator object to the ResNet-50 model to extract the features.

batch_size = 64

root_dir = '101_ObjectCategories'



img_gen = ImageDataGenerator(preprocessing_function=preprocess_input)



datagen = img_gen.flow_from_directory(root_dir,

target_size=(img_size, img_size),

batch_size=batch_size,

class_mode=None,

shuffle=False)



num_images = len(datagen.filenames)

num_epochs = int(math.ceil(num_images / batch_size))



feature_list = model.predict_generator(datagen, num_epochs) print("Num images = ", len(datagen.classes))

print("Shape of feature_list = ", feature_list.shape) Num images = 8677

Shape of feature_list = (8677, 2048)

We have 8677 images in our dataset after removing the google background images. Each image is converted into a 2048 feature vector.

# Get full path for all the images in our dataset

filenames = [root_dir + '/' + s for s in datagen.filenames]

We will fit the nearest neighbor algorithm to the extracted features. We have picked ball_tree algorithm as it works well with sparse data.

neighbors = NearestNeighbors(n_neighbors=5,

algorithm='ball_tree',

metric='euclidean')

neighbors.fit(feature_list)

Let’s pick a new image that is not present in our dataset and try to find similar images for it.

from google.colab import files

uploaded = files.upload()

Once the image is uploaded extract the features for the image using the ResNet-50 and find the nearest neighbors for it.



# ref # ref https://datascience.stackexchange.com/questions/31167/how-to-predict-an-image-using-saved-model img_path = 'ant.jpg'

input_shape = (img_size, img_size, 3)

img = image.load_img(img_path, target_size=(input_shape[0], input_shape[1]))

img_array = image.img_to_array(img)

expanded_img_array = np.expand_dims(img_array, axis=0)

preprocessed_img = preprocess_input(expanded_img_array) test_img_features = model.predict(preprocessed_img, batch_size=1)



_, indices = neighbors.kneighbors(test_img_features)

Helper function to plot the images

def similar_images(indices):

plt.figure(figsize=(15,10), facecolor='white')

plotnumber = 1

for index in indices:

if plotnumber<=len(indices) :

ax = plt.subplot(2,4,plotnumber)

plt.imshow(mpimg.imread(filenames[index]), interpolation='lanczos')

plotnumber+=1

plt.tight_layout()

With the nearest neighbors indices provided by the model let’s find out the images similar to the one we uploaded.

print(indices.shape)



plt.imshow(mpimg.imread(img_path), interpolation='lanczos')

plt.xlabel(img_path.split('.')[0] + '_Original Image',fontsize=20)

plt.show()

print('********* Predictions ***********')

similar_images(indices[0])

Here we observe the image of a scorpion in the similarity images which is incorrect. Due to the large dimensionality of the features, the NearestNeighbors algorithm is not able to cluster similar features accurately.

To overcome this we will apply PCA to our features and reduce the dimensions and try to find similar features again.

pca = PCA(n_components=100)

pca.fit(feature_list)

compressed_features = pca.transform(feature_list)

Fit the nearest neighbors algorithm to the new features.

neighbors_pca_features = NearestNeighbors(n_neighbors=5,

algorithm='ball_tree', metric='euclidean') neighbors_pca_features.fit(compressed_features)

Transform the test image features as well and try to find similar images.

test_img_compressed = pca.transform(test_img_features)

distances, indices = neighbors_pca_features.kneighbors(test_img_compressed)

print(indices.shape)

plt.imshow(mpimg.imread(img_path), interpolation='lanczos')

plt.xlabel(img_path.split('.')[0] + '_Original Image',fontsize=20)

plt.show()

print('********* Predictions ***********')

similar_images(indices[0])

Great! Decreasing the dimensions of the feature helped us in getting better results.

What next ?

Try different pre-trained models like mobilenet,vgg16 etc. and check the accuracy.

Try data augmentation on the dataset.

We have used n_componets in PCA is 100. We can try changing these numbers and see how it affects our predictions.

Try cosine distance metrics for the nearest neighbor algorithm.

Note : As each class have only 8–20 images, accuracy may not be uniform accross all classes. Please try the model on a large dataset to get optimal accuracy.

Please let me know if you encounter any issue or unable to follow the blog.I am always open to feedback. Cheers ..!

Code is available here

Connect with me on Linkedin

Based on my learning from the book