Image Matching with OpenCV’s Template Matching | by Aviad Atlas | Towards Data Science

As a data scientist at VATBox, I’ve mainly worked on projects which at their core involve building Machine Learning models. What’s nice about this project is that it purely includes building an algorithm to solve the given task. A real-world problem with a custom-made solution.
The problem in hand is essentially matching between two images which are close to being identical (changes can be due to image size, for example). The goal here is to do so in real-time, therefore we need the algorithm to be relatively fast.
At VATBox our world is a world of invoices. Users upload reports, which contain images of invoices, to our platform. A single report contains two groups of images, one is a collection of separate invoices, and the other is all the collection together in one single file. For reasons which we will not go into, there may be images in one group that won’t appear in the other. The goal is to detect the images which don’t appear in the other group (if there are any). If we match between all the images which are essentially the same image, we can also identify the spare ones.
So to find the spare images, we need to perform a comparison between every two images (or do we..? we’ll get to that..), one image being from the first group and the other from the second group. Once that is done, we can match between images, and then see which images are left without any match. We can build a Comparison Matrix which holds these comparisons.
During my research, I mostly encountered solutions that are based on features extracted from images to match between images, but that’s computationally expensive for our case, as we’re interested in doing this for every pair of images. Therefore we are in need of a different approach.
As described, the images that actually match, are close to being identical. So we can expect their pixel values to be relatively similar. So can we perhaps calculate the correlation between the two given signals? But how can this be done efficiently with images?
OpenCV has a Template Matching module. The purpose of this module is to find a given template within a (larger) image. The module enables us to “swipe” a template (T) across an image (I) and perform calculations efficiently (similarly to how a convolutional kernel is swiped on an image in a CNN).
This can be great for our needs! We can use this to swipe an image from the first group on an image of the other group efficently, as OpenCV’s implementation is optimal. Of course, since we’re looking for two images that match (and not a template in another image) the swiping is just done a single time.
The Template Matching module includes different methods of calculating correlations, one of them being a normalized calculation, returning values between 0 (no similarity) to 1 (completely identical images):
corr = cv2.matchTemplate(group1_image, group2_image,                         cv2.TM_CCOEFF_NORMED)[0][0]
The indexing shows that we’re accessing a matrix. As mentioned, we’re essentially performing a single swipe, therefore we’re just grabbing the single value from a matrix with a single element.
A word about the calculation. Prior to the actual correlation calculation, the images are normalized, resulting in a matrix (or tensor) with negative values as well as positive values. When performing the correlation calculation, similar regions will get positive values within the correlation calculation, whereas dissimilar regions will get negative values (resulting in lower correlation).
Since we’re comparing every pair of images, the complexity of the algorithm is O(n*m), n being the number of images in group one, and m being the number of images in group two. Can we make the algorithm more efficient?
As the above function returns values close to 1 for similar images, we can set a threshold (e.g. 0.9) for which there’s no need to keep on searching for a match if one is found. In addition, if a match is found, we’ll remove the matched images from our search. For example, if in the first iteration we are comparing an image from group one with 4 images from group two, on the second iteration we’ll do a comparison with only 3 images, and so on.
In this article, we reviewed a somewhat out of the ordinary real-world data science problem which required a tailor-made solution. We dug into OpenCV’s Template Matching and understood how it can be used to fulfill the requirements of the problem at hand. Hope you enjoyed it!