Different Colorspaces as Inputs to CNNs
By: Vardan Agarwal
Tags: Artificial Intelligence, Computer Vision, Towards Data Science, Opencv, TensorFlow

Requirements

If you want to code along, then you would require Tensorflow and OpenCV. You can pip install them using the code shown below, or you use Google Colab like me, where no set up will be required along with free GPU.

pip install tensorflow

pip install opencv-python

Colorspaces Used

Photo by Efe Kurnaz on Unsplash

Deciding which colorspaces to use was a pretty easy task. I just opened the documentation for OpenCV and choose all the unique ones in which it was possible to convert. I will just list them and provide additional reading links if anyone is interested in knowing more about them.

Code and Results

Before beginning, I want to be clear that my aim was not to create a very high-tech state of the art CNN architecture to get the best accuracy but to compare all the colorspaces.

Imports and loading the dataset

The dataset chosen was the cats vs dogs one. First, we start we importing all the required libraries.

import tensorflow as tf

from tensorflow.keras.models import Sequential

from tensorflow.keras.layers import Conv2D, MaxPooling2D, Dropout, Flatten, Dense, Activation, BatchNormalization

import os

import numpy as np

import matplotlib.pyplot as plt

import re

import random

import cv2

The next part is to load the dataset and store all the filenames which will then be passed to the image data generator and define all the constants which will be used.

Creating an image data generator

A normal custom data generator is created with an extra argument of colorspace, which defines which colorspace to convert to. cv2.cvtColor is used for that purpose. The images are resized to their required sizes and normalized by dividing by 255 for all except for the Hue matrice of HSV which is divided by 180. NumPy arrays are created for the images and labels and they are yielded.

A basic CNN is created using Conv2D, max pooling, batch normalization, and dropout layers which are eventually flattened, a dense layer gives output with a sigmoid activation function. The model is compiled using Adam. If you want to create advanced CNN’s which may give better results, you can refer to my previous article. Just fit the model with the required values, and we are done.

Results with normal CNN

If we observe the training accuracy and loss all the colorspaces except HSV give better results which are pretty similar. However, it can also be observed that our model is overfitting so HSV can not just be discredited straightaway. Also, for the validation set, the images in HSV can learn much faster than their counterparts, but for a large number of epochs the results even out and choosing a particular one is very difficult.

Results with Transfer Learning

Again as stated above the transfer learning model used is a pretty simple one. A pre-trained MobileNetV2 is used whose layers are set to non-trainable and a global average pooling layer is followed by a dense layer for output.

Results with non-trainable layers in transfer learning

Only BGR and XYZ colorspaces give good results while HSV gives the poorest results. But why did this happen? Well we use pre-trained layers that were accustomed to seeing RGB style inputs and XYZ is pretty similar to RGB so only they give good results, whereas HSV is a cylindrical system and is the farthest off RGB in terms of similarity, hence gave the worst results.

So we make the whole pre-trained network trainable by changing base_model.trainable = True and let’s see what happens now.

Result with transfer learning with all trainable layers.

Even now BGR and XYZ colorspaces perform great from the start however, they are caught up by all the other colorspaces except HSV which again performed the worst. There is hardly much to choose from among all the colorspaces except HSV so let’s create an ensemble model and see whether that improves performance.

Creating an Ensemble Model

If we can achieve an improvement in performance then we will also know that different colorspaces were getting different images classified as right or wrong, which would mean that changing the colorspace had some impact on the models. We will create a pretty simple ensemble model by taking the mean of the predicted probabilities of all the models, converting it to an integer, and evaluating it against the true labels.