Title: Comparing images for similarity using siamese networks, Keras, and TensorFlow - PyImageSearch

URL Source: https://pyimagesearch.com/2020/12/07/comparing-images-for-similarity-using-siamese-networks-keras-and-tensorflow/

Published Time: 2020-12-07T15:00:00+00:00

Markdown Content:
In this tutorial, you will learn how to compare two images for similarity (and whether or not they belong to the same or different classes) using siamese networks and the Keras/TensorFlow deep learning libraries.

![Image 1](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20700%20450'%3E%3C/svg%3E)

This blog post is part three in our three-part series on the basics of siamese networks:

*   **Part #1:** _[Building image pairs for siamese networks with Python](https://pyimagesearch.com/2020/11/23/building-image-pairs-for-siamese-networks-with-python/)_ (post from two weeks ago)
*   **Part #2:** _[Training siamese networks with Keras, TensorFlow, and Deep Learning](https://pyimagesearch.com/2020/11/30/siamese-networks-with-keras-tensorflow-and-deep-learning/)_ (last week’s tutorial)
*   **Part #3:** _Comparing images using siamese networks_ (this tutorial)

Last week we learned how to train our siamese network. Our model performed well on our test set, correctly verifying whether two images belonged to the _same_ or _different_ classes. After training, we serialized the model to disk.

Soon after last week’s tutorial published, I received an email from PyImageSearch reader Scott asking:

> _“Hi Adrian — thanks for these guides on siamese networks. I’ve heard them mentioned in deep learning spaces but honestly was never really sure how they worked or what they did. This series really helped clear my doubts and have even helped me in one of my work projects._
> 
> _My question is:_
> 
> **_How do we take our trained siamese network and make predictions on it from images outside of the training and testing set?_**
> 
> _Is that possible?_”

You bet it is, Scott. And that’s exactly what we are covering here today.

**To learn how to compare images for similarity using siamese networks, _just keep reading._**

![Image 2](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20128%20128'%3E%3C/svg%3E)

#### Looking for the source code to this post?

[Jump Right To The Downloads Section](#download-the-code)

In the first part of this tutorial, we’ll discuss the basic process of how a _trained_ siamese network can be used to predict the similarity between two image pairs and, more specifically, whether the two input images belong to the _same_ or _different_ classes.

You’ll then learn how to configure your development environment for siamese networks using Keras and TensorFlow.

Once your development environment is configured, we’ll review our project directory structure and then implement a Python script to compare images for similarity using our siamese network.

We’ll wrap up this tutorial with a discussion of our results.

### **How can siamese networks predict similarity between image pairs?**

![Image 3](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20700%20420'%3E%3C/svg%3E)

**Figure 1:** Using siamese networks to compare two images for similarity results in a similarity score. The closer the score is to “1”, the _more similar_ the images are (and are thus more likely to belong to the _same class_). Conversely, the closer the score is to “0”, the _less similar_ the two images are.

In [last week’s tutorial](https://pyimagesearch.com/2020/11/30/siamese-networks-with-keras-tensorflow-and-deep-learning/) you learned how to train a siamese network to verify whether two pairs of digits belonged to the _same_ or _different_ classes. We then serialized our siamese model to disk after training.

The question then becomes:

> _“How can we use our trained siamese network to predict the **similarity** between two images?”_

The answer is that we utilize the final layer in our siamese network implementation, which is **sigmoid activation function.**

The sigmoid activation function has an output in the range _\[0, 1\]_, meaning that when we present an image pair to our siamese network, the model will output a value _\>= 0_ and _<= 1._

A value of `0` means that the two images are _completely and totally dissimilar,_ while a value of `1` implies that the images are _very similar._

An example of such a similarity can be seen in **Figure 1** at the top of this section:

*   Comparing a _“7”_ to a _“0”_ has a low similarity score of only 0.02.
*   However, comparing a _“0”_ to another _“0”_ has a very high similarity score of 0.93.

A good rule of thumb is to use a similarity cutoff value of `0.5` (50%) as your threshold:

*   If two image pairs have an image similarity of _<= 0.5,_ then they belong to a _different_ class.
*   Conversely, if pairs have a predicted similarity of _\> 0.5,_ then they belong to the _same_ class.

In this manner you can use siamese networks to (1) compare images for similarity and (2) determine whether they belong to the same class or not.

Practical use cases of using siamese networks include:

*   **Face recognition:** Given two separate images containing a face, determine if it’s the _same_ person in both photos.
*   **Signature verification:** When presented with two signatures, determine whether one is a forgery or not.
*   **Prescription pill identification:** Given two prescription pills, determine whether they are the same medication or different medications.

### **Configuring your development environment**

This series of tutorials on siamese networks utilizes Keras and TensorFlow. If you intend on following this tutorial or the previous two parts in this series, I suggest you take the time now to configure your deep learning development environment.

You can utilize either of these two guides to install TensorFlow and Keras on your system:

*   _[How to install TensorFlow 2.0 on Ubuntu](https://pyimagesearch.com/2019/12/09/how-to-install-tensorflow-2-0-on-ubuntu/)_
*   _[How to install TensorFlow 2.0 on macOS](https://pyimagesearch.com/2019/12/09/how-to-install-tensorflow-2-0-on-macos/)_

Either tutorial will help you configure your system with all the necessary software for this blog post in a convenient Python virtual environment.

### **Having problems configuring your development environment?**

[![Image 4: Screenshot of PyImageSearch Plus and Google Colab Notebook with Jupyter logo overlaid](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20500%20334'%3E%3C/svg%3E)](https://pyimagesearch.com/pyimagesearch-plus/)

**Figure 2:** Having trouble configuring your dev environment? Want access to pre-configured Jupyter Notebooks running on Google Colab? Be sure to join [PyImageSearch Plus](https://pyimagesearch.com/pyimagesearch-plus/) —- you’ll be up and running with this tutorial in a matter of minutes.

All that said, are you:

*   Short on time?
*   Learning on your employer’s administratively locked system?
*   Wanting to skip the hassle of fighting with the command line, package managers, and virtual environments?
*   **Ready to run the code _right now_ on your Windows, macOS, or Linux system?**

Then join [PyImageSearch Plus](https://pyimagesearch.com/pyimagesearch-plus/) today!

**Gain access to Jupyter Notebooks for this tutorial and other PyImageSearch guides that are _pre-configured_ to run on Google Colab’s ecosystem right in your web browser!** No installation required.

And best of all, these Jupyter Notebooks will run on **Windows, macOS, and Linux!**

### **Project structure**

Before we get too far into this tutorial, let’s first take a second and review our project directory structure.

Start by making sure you use the **_“Downloads”_** section of this tutorial to download the source code and example images.

From there, let’s take a look at the project:

$ tree . --dirsfirst
.
├── examples
│   ├── image\_01.png
...
│   └── image\_13.png
├── output
│   ├── siamese\_model
│   │   ├── variables
│   │   │   ├── variables.data-00000-of-00001
│   │   │   └── variables.index
│   │   └── saved\_model.pb
│   └── plot.png
├── pyimagesearch
│   ├── config.py
│   ├── siamese\_network.py
│   └── utils.py
├── test\_siamese\_network.py
└── train\_siamese\_network.py

4 directories, 21 files

Inside the `examples` directory we have a number of example digits:

![Image 5](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20640%20378'%3E%3C/svg%3E)

**Figure 3:** Examples of digits we’ll be comparing for similarity using siamese networks implemented with Keras and TensorFlow.

We’ll be sampling pairs of these digits and then comparing them for similarity using our siamese network.

The `output` directory contains the training history plot (`plot.png`) and our trained/serialized siamese network model (`siamese_model/`). Both of these files were generated in [last week’s tutorial](https://pyimagesearch.com/2020/11/30/siamese-networks-with-keras-tensorflow-and-deep-learning/) on training your own custom siamese network models — **make sure you read that tutorial before you continue, as it’s _required reading_ for today!**

The `pyimagesearch` module contains three Python files:

1.  `` `config.py` ``: Our configuration file storing important variables such as output file paths and training configurations (including image input dimensions, batch size, epochs, etc.)
2.  `` `siamese_network.py` ``: Our implementation of our siamese network architecture
3.  `` `utils.py` ``: Contains helper configuration functions to generate image pairs, compute Euclidean distances, and plot training history path

The `train_siamese_network.py` script:

1.  Imports the configuration, siamese network implementation, and utility functions
2.  Loads the MNIST dataset from disk
3.  Generates image pairs
4.  Creates our training/testing dataset split
5.  Trains our siamese network
6.  Serializes the trained siamese network to disk

**I will _not_ be covering these four scripts today, as I have _already_ covered them in [last week’s tutorial on how to train siamese networks](https://pyimagesearch.com/2020/11/30/siamese-networks-with-keras-tensorflow-and-deep-learning/)**. I’ve _included_ these files in the project directory structure for today’s tutorial as a matter of completeness, but again, for a full review of these files, what they do, and how they work, refer back to last week’s tutorial.

Finally, we have the focus of today’s tutorial, `test_siamese_network.py`.

This script will:

1.  Load our trained siamese network model from disk
2.  Grab the paths to the sample digit images in the `examples` directory
3.  Randomly construct pairs of images from these samples
4.  Compare the pairs for similarity using the siamese network

Let’s get to work!

### **Implementing our siamese network image similarity script**

We are now ready to implement siamese networks for image similarity using Keras and TensorFlow.

Start by making sure you use the **_“Downloads”_** section of this tutorial to download the source code, example images, and pre-trained siamese network model.

From there, open up `test_siamese_network.py`, and follow along:

\# import the necessary packages
from pyimagesearch import config
from pyimagesearch import utils
from tensorflow.keras.models import load\_model
from imutils.paths import list\_images
import matplotlib.pyplot as plt
import numpy as np
import argparse
import cv2

We start off by importing our required Python packages (**Lines 2-9**). Notable imports include:

*   `` `config` ``: Contains important configurations, including the path to our trained/serialized siamese network model residing on disk
*   `` `utils` ``: Contains the `euclidean_distance` function utilized in our `Lambda` layer of the siamese network — **we need to import this package to suppress any** `UserWarnings` **about loading** `Lambda` **layers from disk**
*   `` `load_model` ``: The Keras/TensorFlow function used to load our trained siamese network from disk
*   `` `list_images` ``: Grabs the paths to all images in our `examples` directory

Let’s move on to parsing our command line arguments:

\# construct the argument parser and parse the arguments
ap = argparse.ArgumentParser()
ap.add\_argument("-i", "--input", required=True,
	help="path to input directory of testing images")
args = vars(ap.parse\_args())

We only need a single argument here, `--input`, which is the path to our directory on disk containing the images we want to compare for similarity. When running this script, we’ll supply the path to the `examples` directory in our project.

With our command line arguments parsed, we can now grab all `testImagePaths` in our `--input` directory:

\# grab the test dataset image paths and then randomly generate a
# total of 10 image pairs
print("\[INFO\] loading test dataset...")
testImagePaths = list(list\_images(args\["input"\]))
np.random.seed(42)
pairs = np.random.choice(testImagePaths, size=(10, 2))

# load the model from disk
print("\[INFO\] loading siamese model...")
model = load\_model(config.MODEL\_PATH)

**Line 20** grabs the paths to all of our example images containing digits we want to compare for similarity. **Line 22** randomly generates a total of 10 pairs of images from these `testImagePaths`.

**Line 26** loads our siamese network from disk using the `load_model` function.

With the siamese network loaded from disk, we can now compare images for similarity:

\# loop over all image pairs
for (i, (pathA, pathB)) in enumerate(pairs):
	# load both the images and convert them to grayscale
	imageA = cv2.imread(pathA, 0)
	imageB = cv2.imread(pathB, 0)

	# create a copy of both the images for visualization purpose
	origA = imageA.copy()
	origB = imageB.copy()

	# add channel a dimension to both the images
	imageA = np.expand\_dims(imageA, axis=-1)
	imageB = np.expand\_dims(imageB, axis=-1)

	# add a batch dimension to both images
	imageA = np.expand\_dims(imageA, axis=0)
	imageB = np.expand\_dims(imageB, axis=0)

	# scale the pixel values to the range of \[0, 1\]
	imageA = imageA / 255.0
	imageB = imageB / 255.0

	# use our siamese model to make predictions on the image pair,
	# indicating whether or not the images belong to the same class
	preds = model.predict(\[imageA, imageB\])
	proba = preds\[0\]\[0\]

**Line 29** starts a loop over all image pairs. For each image pair we:

*   Load the two images from disk (**Lines 31 and 32**)
*   Clone the two images such that we can draw/visualize them later (**Lines 35 and 36**)
*   Add a channel dimension (**Lines 39 and 40**) along with a batch dimension (**Lines 43 and 44**)
*   Scale the pixel intensities to from the range _\[0, 255\]_ to _\[0, 1\]_, just like we did when training our [siamese network last week](https://pyimagesearch.com/2020/11/30/siamese-networks-with-keras-tensorflow-and-deep-learning/) (**Lines 47 and 48**)

Once `imageA` and `imageB` are preprocessed, we compare them for similarity by making a call to the `.predict` method on our siamese network model (**Line 52**), resulting in the probability/similarity scores of the two images (**Line 53**).

The final step is to display the image pair and corresponding similarity score to our screen:

	# initialize the figure
	fig = plt.figure("Pair #{}".format(i + 1), figsize=(4, 2))
	plt.suptitle("Similarity: {:.2f}".format(proba))

	# show first image
	ax = fig.add\_subplot(1, 2, 1)
	plt.imshow(origA, cmap=plt.cm.gray)
	plt.axis("off")

	# show the second image
	ax = fig.add\_subplot(1, 2, 2)
	plt.imshow(origB, cmap=plt.cm.gray)
	plt.axis("off")

	# show the plot
	plt.show()

**Lines 56 and 57** create a `matplotlib` figure for the pair and display the similarity score as the title of the plot.

**Lines 60-67** plot each of the images in the pair on the figure, while **Line 70** displays the output to our screen.

Congrats on implementing siamese networks for image comparison and similarity! Let’s see the results of our hard work in the next section.

### **Image similarity results using siamese networks with Keras and TensorFlow**

We are now ready to compare images for similarity using our siamese network!

Before we examine the results, make sure you:

1.  Have read our [previous tutorial on training siamese networks](https://pyimagesearch.com/2020/11/30/siamese-networks-with-keras-tensorflow-and-deep-learning/) so you understand how our siamese network model was trained and generated
2.  Use the **_“Downloads”_** section of this tutorial to download the source code, pre-trained siamese network, and example images

From there, open up a terminal, and execute the following command:

$ python test\_siamese\_network.py --input examples
\[INFO\] loading test dataset...
\[INFO\] loading siamese model...

![Image 6](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20700%20875'%3E%3C/svg%3E)

**Figure 4:** The results of comparing images for similarity using siamese networks and the Keras/TensorFlow deep learning libraries.

_**Note:** Are you getting an error related to `TypeError: ('Keyword argument not understood:', 'groups')`? If so, keep in mind that the **pre-trained** model included in the “Downloads” section of this tutorial was trained using **TensorFlow 2.3.** You should therefore be using TensorFlow 2.3 when running `test_siamese_network.py`. **If you instead prefer to use a different version of TensorFlow, simply run** `train_siamese_network.py` **to train the model and generate a new** `siamese_model` **serialized to disk.** From there you’ll be able to run `test_siamese_network.py` without error._

**Figure 4** above displays a montage of our image similarity results.

For the first image pair, one contains a _“7”_, while the other contains a _“1”_ — clearly these are not the same image, and the similarity score is low at 42%. **Our siamese network has _correctly_ marked these images as belonging to _different_ classes.**

The next image pair consists of two _“0”_ digits. Our siamese network has predicted a _very high_ similarity score of 97%, **indicating that these two images belong to the _same_ class.**

You can see the same pattern for all other image pairs in **Figure 4**. Images that have a high similarity score belong to the _same_ class, while image pairs with low similarity scores belong to _different_ classes.

Since we used the sigmoid activation layer as the final layer in our siamese network (which has an output value in the range _\[0, 1\]_), a good rule of thumb is to use a similarity cutoff value of `0.5` (50%) as your threshold:

*   If two image pairs have an image similarity of _<= 0.5_, then they belong to _different_ classes.
*   Conversely, if pairs have a predicted similarity of _\> 0.5_, then they belong to the _same_ class.

You can use this rule of thumb in your own projects when using siamese networks to compute image similarity.

### What's next? We recommend [PyImageSearch University](https://pyimagesearch.com/pyimagesearch-university/?utm_source=blogPost&utm_medium=bottomBanner&utm_campaign=What%27s%20next%3F%20I%20recommend).

**Course information:**  
84 total classes • 114+ hours of on-demand code walkthrough videos • Last updated: February 2024  
★★★★★ 4.84 (128 Ratings) • 16,000+ Students Enrolled

**I strongly believe that if you had the right teacher you could _master_ computer vision and deep learning.**

Do you think learning computer vision and deep learning has to be time-consuming, overwhelming, and complicated? Or has to involve complex mathematics and equations? Or requires a degree in computer science?

That’s _not_ the case.

All you need to master computer vision and deep learning is for someone to explain things to you in _simple, intuitive_ terms. _And that’s exactly what I do_. My mission is to change education and how complex Artificial Intelligence topics are taught.

If you're serious about learning computer vision, your next stop should be PyImageSearch University, the most comprehensive computer vision, deep learning, and OpenCV course online today. Here you’ll learn how to _successfully_ and _confidently_ apply computer vision to your work, research, and projects. Join me in computer vision mastery.

**Inside PyImageSearch University you'll find:**

*   ✓ **84 courses** on essential computer vision, deep learning, and OpenCV topics
*   ✓ **84 Certificates** of Completion
*   ✓ **114+ hours** of on-demand video
*   ✓ **Brand new courses released _regularly_**, ensuring you can keep up with state-of-the-art techniques
*   ✓ **Pre-configured Jupyter Notebooks in Google Colab**
*   ✓ Run all code examples in your web browser — works on Windows, macOS, and Linux (no dev environment configuration required!)
*   ✓ Access to **centralized code repos for _all_ 536+ tutorials** on PyImageSearch
*   ✓ **Easy one-click downloads** for code, datasets, pre-trained models, etc.
*   ✓ **Access** on mobile, laptop, desktop, etc.

[Click here to join PyImageSearch University](https://pyimagesearch.com/pyimagesearch-university/?utm_source=blogPost&utm_medium=bottomBanner&utm_campaign=What%27s%20next%3F%20I%20recommend)

**Summary**
-----------

In this tutorial you learned how to compare two images for similarity and, more specifically, whether they belonged to the _same_ or _different_ classes. We accomplished this task using siamese networks along with the Keras and TensorFlow deep learning libraries.

This post is the final part in our three part series on introduction to siamese networks. For easy reference, here are links to each guide in the series:

*   **Part #1: _[Building image pairs for siamese networks with Python](https://pyimagesearch.com/2020/11/23/building-image-pairs-for-siamese-networks-with-python/)_**
*   **Part #2: _[Training siamese networks with Keras, TensorFlow, and Deep Learning](https://pyimagesearch.com/2020/11/30/siamese-networks-with-keras-tensorflow-and-deep-learning/)_**
*   **Part #3:** _Comparing images for similarity using siamese networks, Keras, and TensorFlow_ (this tutorial)

In the near future I’ll be covering more advanced series on siamese networks, including:

*   Image triplets
*   Contrastive loss
*   Triplet loss
*   Face recognition with siamese networks
*   One-shot learning with siamese networks

Stay tuned for these tutorials; you don’t want to miss them!

**To download the source code to this post (and be notified when future tutorials are published here on PyImageSearch), _simply enter your email address in the form below!_**

![Image 7](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20410%20567'%3E%3C/svg%3E)

#### Download the Source Code and FREE 17-page Resource Guide

Enter your email address below to get a .zip of the code and a **FREE 17-page Resource Guide on Computer Vision, OpenCV, and Deep Learning.** Inside you'll find my hand-picked tutorials, books, courses, and libraries to help you master CV and DL!