Switching among OpenCV, Tensorflow and Pillow? Wait!!! | by Saksham Sinha | Towards Data Science

Modern Computer Vision (CV) is currently a hot field of research which involves largely working with images. For this, one needs to use a framework to open those images to do some processing on them.
In today’s rapid development of frameworks, every framework has its own way of handling images, each with its own specifications. Therefore, CV solutions developed in one framework may not work as expected in the other framework. The blog “How Tensorflow’s tf.image.resize stole 60 days of my life” (https://hackernoon.com/how-tensorflows-tf-image-resize-stole-60-days-of-my-life-aba5eb093f35) is a perfect example of this type of situation. It can take up multiple days to figure out what went wrong and could delay the project extensively.
This blog will discuss one such use case in detail where OpenCV and Tensorflow show the differences in reading and resizing a JPEG image. It will also show a way to make them work consistently. Also, thanks to Tejas Pandey for the help in achieving consistency between OpenCV and Tensorflow.
We’ll start with opening a simple JPEG image of a dog with two frameworks Tensorflow 1.x and OpenCV.
Let’s get started. We’ll import a few essential libraries in python.
%tensorflow_version 1.ximport cv2import matplotlib.pyplot as pltimport numpy as npfrom skimage import ioimport tensorflow as tf
Let’s open an image using OpenCV.
image_path = 'dog2.JPG'def plt_display(image, title):  fig = plt.figure()  a = fig.add_subplot(1, 1, 1)  imgplot = plt.imshow(image)  a.set_title(title)image = cv2.imread(image_path)plt_display(image, 'cv2-BGR')
Something’s strange about this figure above. It doesn’t look like the original. Well, yes, that’s true. Before diving into this, first let’s learn how colors in images are represented. Generally, images have three colors channels Red, Green, and Blue (RGB) which produces the colors in the pixel. The order of these channels changes the color of the pixels, as pixels will always interpret first channel as Red, second channel as Green and third channel as Blue.
Now that we have an understanding of how color channels work in an image, let’s look into why we saw a different color image above. When we open an image using OpenCV, by default OpenCV opens the image in Blue, Green and Red channel (BGR). This is fine but when we display the image, the pixels misinterpret the channels (i.e. pixels get confused and interpret Blue as Red and vice-versa). That’s why we get the above wrong colors in the image.
So how to fix this? After opening the image, we convert the image from BGR to RGB. We do this by using the OpenCV flag COLOR_BGR2RGB. The code below shows how to open an image with OpenCV correctly.
image_cv = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)print(image_cv.dtype)print(np.max(image_cv))plt_display(image_cv, 'cv2-RGB')
Let’s open an image with Tensorflow now and see if we get the same results. It’s a two-step process to open a JPEG image in Tensorflow. The first step is to open the image, and the second step is to perform JPEG decoding as JPEG is a compressed image.
print("Tensorflow version: ", tf.__version__)image_tf = tf.io.read_file(image_path)image_tf = tf.image.decode_jpeg(image_tf, channels=3)with tf.Session() as sess:    image_tf = image_tf.eval()print(image_tf.dtype)plt_display(image_tf, 'TF')
On visual inspection, the image read from OpenCV and Tensorflow look the same. But are they? To see what are the differences, let’s subtract the two. If the two are same, we should see a completely black image. If not, then we will see some pixels with different colors.
image_diff = np.abs(image_cv- image_tf)plt_display(image_diff, 'OpenCV-TF')
Woah! That’s a huge difference. What went wrong? Well, this difference is arising from the fact that OpenCV, by default, uses integer accurate decompression of the JPEG image. In contrast, TensorFlow uses Discrete Cosine Transform as default. This type of decoding is inaccurate and so to make it the same as OpenCV, we need to decode it by using integer accurate decompression. This can be done by setting the parameter dct_method=’INTEGER_ACCURATE’ as shown below.
image_tf = tf.io.read_file(image_path)image_tf = tf.image.decode_jpeg(image_tf, channels=3, dct_method='INTEGER_ACCURATE')with tf.Session() as sess:    image_tf = image_tf.eval()plt_display(image_tf,'TF_INT_ACC')
And now subtracting with the OpenCV image we get a black image. This means both our TF and OpenCV are now consistent in reading the image. This is an essential step because a small change in how you read the image will give rise to significant differences in the processing of the image later.
image_diff = np.abs(image_cv- image_tf)plt_display(image_diff, 'OpenCV-TF')
Now, since we have read the image consistently with OpenCV and TensorFlow, let’s try the resizing of the images with these frameworks. We’ll begin with OpenCV resize first.
# resizing the image with OpenCVprint("image shape before resize:", image_cv.shape)image_cv_resized = cv2.resize(image_cv,(300,300))print("image shape after resize:", image_cv_resized.shape)print("image dtype: ", image_cv_resized.dtype)
We have the OpenCV resize done. Now let’s do the TensorFlow resize. Tensorflow work with tensors. It’s resizing method expects a 4D tensor and returns a 4D tensor output. Hence, we will have to first expand the image from three to four. Then resize the image and squeeze the dimensions back to three. Since our image is still a tensor, we will create and run a Tensorflow session to get back our resized image in NumPy format. Once we have the NumPy image, we convert it to uint8 type as by default, session converts the NumPy to float32 type. This makes the types inconsistent with OpenCV. This is shown below-
# resizing the image with tensorflow 1.xprint("image shape before resize:", image_tf.shape)print("image dtype: ", image_tf.dtype)#This function takes in a 4D input. Hence we will have to expand the image and then squeeze back to three dimensions before we can use it as an image.image_tf_4D= tf.expand_dims(image_tf,0)# doing bilinear resizeimage_tf_resized_4D = tf.image.resize_bilinear(    image_tf_4D,    (300,300))#squeezing back the image to 3Dimage_tf_resized = tf.squeeze(image_tf_resized_4D)#Above is still a tensor. So we need to convert it to numpy. We do this by using tf session.with tf.Session() as sess:  image_tf_resized = image_tf_resized.eval()print("image shape after resize:", image_tf_resized.shape)print("image dtype: ", image_tf_resized.dtype)#Since it is in float32 format, we need to convert it back to uint8.image_tf_resized = image_tf_resized.astype(np.uint8)print("image dtype after conversion uint8: ", image_tf_resized.dtype)
Now let’s see the difference between the resized images from OpenCV and TF.
image_resized_diff = np.abs(image_cv_resized- image_tf_resized)plt_display(image_resized_diff, 'Resized OpenCV2 - Resized TF')
We again see a big difference in the resize method of the two frameworks. Even when we opened the images consistently across both frameworks, the resize method gave different results. OpenCV resize method, by default, uses bilinear transformation. We used the same bilinear method with Tensorflow. Still, we ended up with different results.
This happened because OpenCV adds half-pixel corrections to the image while resizing. Whereas Tensorflow by default doesn’t. This adds up the difference in the resizing method outputs.
In order to fix this problem, there is a parameter in the TensorFlow bilinear resize that will do the half-pixel correction. This is shown below-
image_tf_4D= tf.expand_dims(image_tf,0)# doing bilinear resize with half pixel correctionimage_tf_resized_hpc_4D = tf.image.resize_bilinear(    image_tf_4D,    (300,300),    half_pixel_centers=True)image_tf_resized_hpc = tf.squeeze(image_tf_resized_hpc_4D)with tf.Session() as sess:  image_tf_resized_hpc = image_tf_resized_hpc.eval()image_tf_resized_hpc = image_tf_resized_hpc.astype(np.uint8)image_resized_diff = np.abs(image_cv_resized- image_tf_resized_hpc)plt_display(image_resized_diff, 'Resized OpenCV2 - Resized TF')
And now we have the expected result. Finally, we have Tensorflow and OpenCV working consistently with reading and resizing images.
A new person in the Computer Vision field trying out these functions will more likely make these mistakes and will be off to a bad start. Tutorials across the internet do not discuss this in detail. I created this blog to show people that from one framework to another, small minor things like what I discussed need to be checked and kept consistent. Otherwise, they will have different results in different frameworks and will adversely affect their work.
Another framework widely used is Pillow (PIL). Let’s see the differences with PIL and OpenCV. We’ll first read the image using PIL.
from PIL import Imageorig_image_pil = Image.open(image_path)image_pil = np.asarray(orig_image_pil)print(image_pil.dtype)plt_display(image_pil, 'PIL')
Now we will see if there is any difference between JPEG image opened by PIL and OpenCV.
image_cv_pil_diff = np.abs(image_cv - image_pil)plt_display(image_cv_pil_diff, 'CV-PIL-DIFF')
As expected from our experience from TF and OpenCV, these two frameworks again open the images differently. PIL, by default, doesn’t provide any way to use integer accurate decompression. However, one can write their own decoder to decode images. But in this blog, I am only showing the differences and if possible, to make them consistent if the library provides the support for it.
Since the images are read differently between these frameworks, this difference will propagate when we do resize. This is shown below-
image_pil_resized = orig_image_pil.resize((300,300),       resample=Image.BILINEAR)image_pil_resized = np.asarray(image_pil_resized)#making sure we have the same dtype as OpenCV image.print(image_pil_resized.dtype)
image_cv_pil_resize_diff = np.abs(image_cv_resized - image_pil_resized)plt_display(image_cv_pil_resize_diff, 'Resized CV- Resized PIL')
Hence, we get the expected difference in resizing. PIL library doesn’t provide any support for half-pixel correction, and so there is no by default support for this. Since we know TF and OpenCV are consistent, we can expect the same results when considering the difference between PIL and TF as well.
Hence, when using PIL framework, one must keep this in mind that one cannot switch directly to another framework if the work is dependent on the pre-processing of the images such as above. This will lead to unexpected results.
I hope that this blog was useful to you and was able to make you aware of the framework dependent processing.