Different Colorspaces as Inputs to CNNs | by Vardan Agarwal | Towards Data Science

I had just finished a rant in another article where I forgot that OpenCV used BGR format and the output was checked in RGB for a CNN and how it had wasted a lot of time when I got an idea, what if passed images in different colorspaces to CNN’s and how will it affect the model? That is what we are going to find out in this article.
Requirements
Requirements
2. Colorspaces Used
3. Code and Results
Imports and loading the dataset
Creating an image data generator
Results with normal CNN
Results with transfer learning
Creating an Ensemble model
If you are not interested in the process and just want to know the results you can jump straight down to the results with the normal CNN section.
If you want to code along, then you would require Tensorflow and OpenCV. You can pip install them using the code shown below, or you use Google Colab like me, where no set up will be required along with free GPU.
pip install tensorflowpip install opencv-python
Deciding which colorspaces to use was a pretty easy task. I just opened the documentation for OpenCV and choose all the unique ones in which it was possible to convert. I will just list them and provide additional reading links if anyone is interested in knowing more about them.
RGB or BGR — Additional reading
HSV — Additional Reading
YCbCr — Additional Reading
LAB — Additional Reading
LUV — Additional Reading
XYZ — Additional Reading
Before beginning, I want to be clear that my aim was not to create a very high-tech state of the art CNN architecture to get the best accuracy but to compare all the colorspaces.
The dataset chosen was the cats vs dogs one. First, we start we importing all the required libraries.
import tensorflow as tffrom tensorflow.keras.models import Sequentialfrom tensorflow.keras.layers import Conv2D, MaxPooling2D, Dropout, Flatten, Dense, Activation, BatchNormalizationimport osimport numpy as npimport matplotlib.pyplot as pltimport reimport randomimport cv2
The next part is to load the dataset and store all the filenames which will then be passed to the image data generator and define all the constants which will be used.
A normal custom data generator is created with an extra argument of colorspace, which defines which colorspace to convert to. cv2.cvtColor is used for that purpose. The images are resized to their required sizes and normalized by dividing by 255 for all except for the Hue matrice of HSV which is divided by 180. NumPy arrays are created for the images and labels and they are yielded.
A basic CNN is created using Conv2D, max pooling, batch normalization, and dropout layers which are eventually flattened, a dense layer gives output with a sigmoid activation function. The model is compiled using Adam. If you want to create advanced CNN’s which may give better results, you can refer to my previous article. Just fit the model with the required values, and we are done.
If we observe the training accuracy and loss all the colorspaces except HSV give better results which are pretty similar. However, it can also be observed that our model is overfitting so HSV can not just be discredited straightaway. Also, for the validation set, the images in HSV can learn much faster than their counterparts, but for a large number of epochs the results even out and choosing a particular one is very difficult.
Again as stated above the transfer learning model used is a pretty simple one. A pre-trained MobileNetV2 is used whose layers are set to non-trainable and a global average pooling layer is followed by a dense layer for output.
Only BGR and XYZ colorspaces give good results while HSV gives the poorest results. But why did this happen? Well we use pre-trained layers that were accustomed to seeing RGB style inputs and XYZ is pretty similar to RGB so only they give good results, whereas HSV is a cylindrical system and is the farthest off RGB in terms of similarity, hence gave the worst results.
So we make the whole pre-trained network trainable by changing base_model.trainable = True and let’s see what happens now.
Even now BGR and XYZ colorspaces perform great from the start however, they are caught up by all the other colorspaces except HSV which again performed the worst. There is hardly much to choose from among all the colorspaces except HSV so let’s create an ensemble model and see whether that improves performance.
If we can achieve an improvement in performance then we will also know that different colorspaces were getting different images classified as right or wrong, which would mean that changing the colorspace had some impact on the models. We will create a pretty simple ensemble model by taking the mean of the predicted probabilities of all the models, converting it to an integer, and evaluating it against the true labels.
accuracy for bgr  : 0.9596773982048035accuracy for ycrcb  : 0.9536290168762207accuracy for lab  : 0.9415322542190552accuracy for luv  : 0.9546371102333069accuracy for xyz  : 0.9546371102333069Ensemble accuracy: 0.966
So to conclude it can be said that changing the colorspace may or may not improve accuracy especially if you are checking randomly and have not assigned a random seed to repeat results because there were a lots of ups and downs. However, if you want just a little more accuracy you can try other colorspaces or even go for ensemble model.
You can find the links to gist link to colab files here and here if you want to play with them.