Data Augmentation and Handling Huge Datasets with Keras: A Simple Way | by Lucas Robinet | Towards Data Science

In today’s tutorial, we will see a well-known technique to reduce overfitting in computer vision and a simple way to implement it : Data Augmentation. We will be going over the following.
A Definition of Data Augmentation
The Keras’ ImageDataGeneratorClass
How to handle Datasets that does not fit into memory
In the Deep Learning field, the performance of a model often improves with the amount of data that has been used to train it. Data Augmentation artificially increases the size of the training set by generating new variant of each training instance. It is a very well known and widespread technique for all computer vision problems that allows the creation of new training instances belonging to the same class as the base instance. Data augmentation forces the model to be more tolerant and it increases its capacity to generalize. It encompasses a wide range of techniques to slightly and randomly disrupt the original data.
The disturbances are mostly translations, distortions, horizontal and vertical flips, changes in brightness, scale, ZCA or PCA whitening...
With this technique, the network trains on slightly different images in each epoch. This technique is only used during training and not during the test phase.
Data Augmentation results in a reduction in overfitting and an improvement in accuracy. On datasets with little data such as 17-flowers for example, using data augmentation can quickly lead to significant improvements in accuracy.
I will describe two methods of data augmentation right away.
The first method is to simply increase the size of your training set. This method consists in loading an image into memory, modifying it, taking the transformed image and the original image, and saving them both in the training file. This method can be useful but does not really help our model to generalize.
The most widespread method is the second method also called On The Fly Data Augmentation [1]. It is this method that the ImageDataGenerator class implements. In this case, the model does not train on the initial instance but only on the generated instance which changes at each epoch to allow the network to better generalize.
Here we are going to talk about this Keras class, see the modifications on the image that we can perform, how to train a model using data augmentation and all the code that goes with it. For all our first experiments, we are going to use this very nice photo from Robert Woeger
This ImageDataGenerator class allows to generate batches of tensor image data with real-time data augmentation. The data will be looped over (in batches).
Since this class is a Python Generator, if you are not familiar with this, I invite you to look at the RealPython tutorial. Now let’s take a closer look at how it works [2] :
Some arguments are self-explanatory but for others it may be interesting to look at their effects and this is what we will do right now. Let’s start doing Data Augmentation!
To perform shifts, the width_shift_range and height_shift_range arguments must be adjusted and set. The value of this parameter is the percentage of shift along the chosen dimension. Let’s look at the code to perform 20% shifts augmentation.
Let’s go through the code together.
We start by importing the necessary packages, here the Keras functions to load an image and convert it into a numpy array format.
Line 9: we extend the dimension of our image, our ImageDataGenerator class expects to receive data with the shape (None, height, width, depth).
The flow() method is then used to generate real-time data augmentation.We also indicate the folder to store the image with the argument save_to_dir and the format for our new training instance (here jpg). The argument save_prefix is used to sign the necessary images by adding this term before the generic name created by the Keras class.
Knowing that imGen is a Python generator, you can generate a new image by calling the next() method. Here we generate 6 new images.
You can easily visualize these new images with the following code.
And here is the result !
A rotation augmentation randomly rotates the image by ± the value in degrees set to the rotation_range parameter. For instance, for a ± 30 degrees rotation, the code looks quite similar
This method allows an image to be augmented by either randomly darkening or brightening it. It is all controlled by the brightness_range argument. Values less than 1.0 darken the image while values larger than 1.0 brighten the image.
You know what happens next.
And the result !
It’s quite complicated to go through all the possibilities of this class. However, here are a few examples [3]:
Vertical and horizontal image flipping: quite self-explanatory. Simply set horizontal_flip and vertical_flip arguments to True.
Image zooming: quite self-explanatory also. The zoom_range argument allows our image to be zoomed by a factor randomly between [1-zoom_range, 1+zoom_range]
Image shearing: controls the maximum angle (in radians) in the counterclockwise direction in which our image will be randomly sheared.
Feature Standardization: this is an important notion that we have not studied yet because it didn’t make sense until now since we were working on a single image. However, it is possible to standardize each pixel across an entire dataset for each feature. To do this you just have to set the featurewise_center and featurewise_std_normalization arguments to True.
ZCA whitening: a whitening transform allows better visualization and understanding of the structure and features of our data. Explaining the ZCA operation in detail is outside the scope of this tutorial. However, if you are interested, feel free to read Learning Multiple Layers of Features from Tiny Images by Alex Krizhevsky.
Whether there is a magic set of arguments that would work in any case is a question you are entitled to ask yourself. Unfortunately, the answer is no, as you expected the Data Augmentation must be performed according to the dataset. So, for example to recognize cars, allowing vertical flip would make no sense since the algorithm will never see flipped cars (unless you are starring in Inception right now, but that’s unlikely). However, if the objective is to know if jellyfish are presents within an image, then vertical flip augmentation remains a useful tool.
However, here is a simple example that you can set up to start your Data Augmentation.
Now that we know how to manipulate the Keras’ ImageDataGenerator class and are familiar with most of the Data Augmentation possibilities, let’s look at how to train a network with real-time Data Augmentation and training instances that change at each epoch. For this part, I took the code from the Keras Documentation [2] which treats the CIFAR-10 dataset.
First, we load the dataset into memory and encode our labels. Then we define our ImageDataGenerator object with the Data Augmentation we want to achieve.
If you want to perform standardization or a ZCA whitening, you have to use the fit method on your generator (see Line 9).
Finally, we use the fit() method of our model with the standard syntax in order to apply image augmentation. The generated images aren’t generated at the same time, they are generated in batches for flexibility (i.e 32 in our case).So at each batch, 32 images will be randomly generated. We also have to define the steps_per_epoch argument, it determines the number of iterations of training per epoch. We set it naturally to (dataset size/batch size) in this case so the network is trained on the same number of images, as there are initially in the dataset, at each epoch.Then the parameters are the same as for classic training.
Sometimes, or rather all the time in modern problems, the dataset is too big to fit into memory.
With the exponential advance in the Deep Learning field, datasets are quite enormous and this growth in GigaBytes results in a very important issue: memory management. Keras’ ImageDataGenerator class allows to implement a solution where images are loaded and augmented in batch as the training progresses. Let’s see how to use it on the Animals Dataset
First, your directories must be organized as follows.
datasets/    animals/        train/            dogs/                dogs_00001.jpg                ...            cats/                cats_00001.jpg                ...            panda/                panda_00001.jpg        val/                dogs/                dogs_00801.jpg                ...            cats/                cats_00801.jpg                ...            panda/                panda_00801.jpg                ...
Now we will use the method flow_from_directory.
The flow_from_directory method takes into account by default the depth of our images. So, in our TARGET_SIZE we put only width and height: the dimensions to which all images found will be resized. This piece of code prints us a message that matches the information on our dataset.
Found 2400 images belonging to 3 classes.Found 600 images belonging to 3 classes.
So, everything seems to be good, it’s normal, it’s good!
N.B: For the Test Generator, no data augmentation is applied, we simply normalize the pixels to be consistent with the data that will be used to train the network.
The training code is quite similar to the one in the previous section.
And that’s it !
In this tutorial we have seen what Data Augmentation is, its usefulness, the main methods and how to implement it with the ImageDataGenerator class. I presented two types of Data Augmentation in the first part. There are of course others, such as the combination of the two methods (widely used in behavioral cloning in particular).
Then, we implement a code to manage a dataset too big to fit into memory. This method is very simple and proves useful in many cases. However, there is a major problem with this method, each image on the disk requires an I/O operation which slows down the training. Adding I/O latency is the last thing you want when you are training a deep neural network that already takes a lot of time.
To overcome this problem, we can for example choose to store our dataset in an HDF5 file and then generate data from this file. This method will be the subject of a future tutorial.
In the meantime, thank you for reading this article, I hope you found it useful.
[1] A. Rosebrock, Keras ImageDataGenerator and Data Augmentation (2019), pyimagesearch
[2] Keras ImageDataGenerator class
[3] Jason Brownlee, Image Augmentation for Deep Learning with Keras (2016), Machine Learning Mastery