Complete Image Augmentation in OpenCV | Towards Data Science

A couple of days ago I was writing an article on using different colorspaces as inputs to CNN’s and for that, I had to use a custom data generator. This meant I could not use the Tensorflow’s inbuilt Image Data Generator for image augmentation. I searched online and found some articles but could not find anything which covered the subject in its entirety so I decided I’ll do one.
Horizontal Shift
Vertical Shift
Brightness
Zoom
Channel Shift
Horizontal Flip
Vertical Flip
Rotation
Fill Mode
If you want to know how any of these works, you can have a look at this article. For visualization, I will be using an image of arc de Triomphe shown below.
Horizontal shift or translation is shifting the image left or right based on a ratio that defines how much maximum to shift. To resize the image back to its original dimensions Keras by default uses a filling mode called ‘nearest’.
I feel this renders that part of that image useless and hence I will simply resize them to their original size. So to perform this we need to generate a random number and then shift the image left or right using indexing.
import cv2import randomimg = cv2.imread('arc_de_triomphe.jpg')def fill(img, h, w):    img = cv2.resize(img, (h, w), cv2.INTER_CUBIC)    return img        def horizontal_shift(img, ratio=0.0):    if ratio > 1 or ratio < 0:        print('Value should be less than 1 and greater than 0')        return img    ratio = random.uniform(-ratio, ratio)    h, w = img.shape[:2]    to_shift = w*ratio    if ratio > 0:        img = img[:, :int(w-to_shift), :]    if ratio < 0:        img = img[:, int(-1*to_shift):, :]    img = fill(img, h, w)    return imgimg = horizontal_shift(img, 0.7)cv2.imshow('Result', img)cv2.waitKey(0)cv2.destroyAllWindows()
After we know how to implement a horizontal shift, it is pretty easy to implement a vertical shift.
import cv2import randomimg = cv2.imread('arc_de_triomphe.jpg')def fill(img, h, w):    img = cv2.resize(img, (h, w), cv2.INTER_CUBIC)    return imgdef vertical_shift(img, ratio=0.0):    if ratio > 1 or ratio < 0:        print('Value should be less than 1 and greater than 0')        return img    ratio = random.uniform(-ratio, ratio)    h, w = img.shape[:2]    to_shift = h*ratio    if ratio > 0:        img = img[:int(h-to_shift), :, :]    if ratio < 0:        img = img[int(-1*to_shift):, :, :]    img = fill(img, h, w)    return imgimg = vertical_shift(img, 0.7)cv2.imshow('Result', img)cv2.waitKey(0)cv2.destroyAllWindows()
This was easy as I had already written an article on creating different Image filters cartooning, summer, winter, etc and it already included brightness function.
medium.com
We utilize HSV colorspace for this task. The more the value of Saturation and Value matrices the greater is the brightness. So to increase brightness, multiply them by a value greater than 1 and to reduce brightness, by a value less than 1. In Tensorflow a range is specified and a random value is chosen between it, so that is how we’ll make it.
import cv2import randomimport numpy as npimg = cv2.imread('arc_de_triomphe.jpg')def brightness(img, low, high):    value = random.uniform(low, high)    hsv = cv2.cvtColor(img, cv2.COLOR_BGR2HSV)    hsv = np.array(hsv, dtype = np.float64)    hsv[:,:,1] = hsv[:,:,1]*value    hsv[:,:,1][hsv[:,:,1]>255]  = 255    hsv[:,:,2] = hsv[:,:,2]*value     hsv[:,:,2][hsv[:,:,2]>255]  = 255    hsv = np.array(hsv, dtype = np.uint8)    img = cv2.cvtColor(hsv, cv2.COLOR_HSV2BGR)    return imgimg = brightness(img, 0.5, 3)cv2.imshow('Result', img)cv2.waitKey(0)cv2.destroyAllWindows()
In Tensorflow, zoom function takes a range and uses random values between it. If those values are less than 1 it magnifies the image and for values greater than 1 it shrinks it and fills it ‘nearest’. However, as we are resizing the images to their size we will only take values less than 1. Example: A value of 0.6 will mean to take 60% of the whole image and then we will resize it back to the original size.
import cv2import randomimg = cv2.imread('arc_de_triomphe.jpg')def fill(img, h, w):    img = cv2.resize(img, (h, w), cv2.INTER_CUBIC)    return imgdef zoom(img, value):    if value > 1 or value < 0:        print('Value for zoom should be less than 1 and greater than 0')        return img    value = random.uniform(value, 1)    h, w = img.shape[:2]    h_taken = int(value*h)    w_taken = int(value*w)    h_start = random.randint(0, h-h_taken)    w_start = random.randint(0, w-w_taken)    img = img[h_start:h_start+h_taken, w_start:w_start+w_taken, :]    img = fill(img, h, w)    return imgimg = zoom(img, 0.5)cv2.imshow('Result', img)cv2.waitKey(0)cv2.destroyAllWindows()
In Channel shift, a random amount of value is added to all the channels of the images chosen from the range specified. Its results are somewhat like the brightness functions only.
import cv2import randomimport numpy as npimg = cv2.imread('arc_de_triomphe.jpg')def channel_shift(img, value):    value = int(random.uniform(-value, value))    img = img + value    img[:,:,:][img[:,:,:]>255]  = 255    img[:,:,:][img[:,:,:]<0]  = 0    img = img.astype(np.uint8)    return imgimg = channel_shift(img, 60)cv2.imshow('Result', img)cv2.waitKey(0)cv2.destroyAllWindows()
I don’t think this requires any explanation. It takes a boolean variable that defines if a horizontal flip is to be performed or not. There is an inbuilt function in OpenCV cv2.flip which is used to perform just this operation.
import cv2img = cv2.imread('arc_de_triomphe.jpg')def horizontal_flip(img, flag):    if flag:        return cv2.flip(img, 1)    else:        return imgimg = horizontal_flip(img, True)cv2.imshow('Result', img)cv2.waitKey(0)cv2.destroyAllWindows()
Like a horizontal flip, it is performed using cv2.flip however, its second argument should be 0.
import cv2img = cv2.imread('arc_de_triomphe.jpg')def vertical_flip(img, flag):    if flag:        return cv2.flip(img, 0)    else:        return imgimg = vertical_flip(img, True)cv2.imshow('Result', img)cv2.waitKey(0)cv2.destroyAllWindows()
To perform a rotation in OpenCV we need to create a rotation matrix and them apply affine transform on it.
import cv2img = cv2.imread('arc_de_triomphe.jpg')def rotation(img, angle):    angle = int(random.uniform(-angle, angle))    h, w = img.shape[:2]    M = cv2.getRotationMatrix2D((int(w/2), int(h/2)), angle, 1)    img = cv2.warpAffine(img, M, (w, h))    return imgimg = rotation(img, 30)    cv2.imshow('Result', img)cv2.waitKey(0)cv2.destroyAllWindows()
Above I had mentioned the ‘nearest’ fill mode and I had stated that I am not going to use it but instead resize the images. I decided I’ll show you how that and all the other fill modes can be made in OpenCV anyway as I was trying to make a complete list, so if you ever want to perform one, you won’t have any problems. From Tensorflow’s documentation:
fill_mode: One of {"constant", "nearest", "reflect" or "wrap"}. Default is 'nearest'. Points outside the boundaries of the input are filled according to the given mode:
‘constant’: kkkkkkkk|abcd|kkkkkkkk (cval=k)
‘nearest’: aaaaaaaa|abcd|dddddddd
‘reflect’: abcddcba|abcd|dcbaabcd
‘wrap’: abcdabcd|abcd|abcdabcd
I’ll make them with a horizontal shift. cv2.copyMakeBorder can be utilized for filling. It takes arguments as image, top, bottom, left, right, and mode. Top, bottom, left, right are the size of the border.
import cv2from matplotlib import pyplot as pltimg = cv2.imread('arc_de_triomphe.jpg')img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)def fill_mode(img, left, right):    nearest = cv2.copyMakeBorder(img, 0, 0, left, right, cv2.BORDER_REPLICATE)    reflect = cv2.copyMakeBorder(img, 0, 0, left, right, cv2.BORDER_REFLECT)    wrap = cv2.copyMakeBorder(img, 0, 0, left, right, cv2.BORDER_WRAP)    constant= cv2.copyMakeBorder(img, 0, 0, left, right, cv2.BORDER_CONSTANT,value=(255, 0, 0))        plt.subplot(221),plt.imshow(nearest,'gray'),plt.title('NEAREST'),plt.axis('off')    plt.subplot(222),plt.imshow(reflect,'gray'),plt.title('REFLECT'),plt.axis('off')    plt.subplot(223),plt.imshow(wrap,'gray'),plt.title('WRAP'),plt.axis('off')    plt.subplot(224),plt.imshow(constant,'gray'),plt.title('CONSTANT'),plt.axis('off')def horizontal_shift_mode(img, ratio):    if ratio > 1 or ratio < 0:        print('Value for horizontal shift should be less than 1 and greater than 0')        return img    ratio = random.uniform(-ratio, ratio)    h, w = img.shape[:2]    to_shift = int(w*ratio)    if ratio > 0:        img = img[:, :w-to_shift, :]        fill_mode(img, to_shift, 0)    if ratio < 0:        img = img[:, -1*to_shift:, :]        fill_mode(img, 0, -1*to_shift)horizontal_shift_mode(img, 0.8)
That’s all from this article. If you want to know how to create a custom data generator then you can have a look at my article below.
towardsdatascience.com
Leonardo Miranda brought to my notice about two awesome libraries called imagaug and albumentations for image augmentation. They are available for both pip and conda install. They provide a large number of image augmentations from basic to very advanced. The best part I found about them was that there were augmentations even for segmentation and bounding boxes. I would encourage everyone to check out their Readme’s they are brilliant.