What is Image Classification? Data Augmentation? Transfer Learning?
By: Christophe Pere
Tags: Image Classification, Towards Data Science, Deep Learning, Data Science, Machine Learning

What is Image Classification? Data Augmentation? Transfer Learning?

This article is the first part of three articles about computer vision. Part 2 will explain Object Recognition. Part 3 will be about Image Segmentation. With this article is provided a notebook: here on GitHub

Introduction

What is more exciting than seeing the world? To be able to see the best around us? The beauty of a sunset, the memorable waterfalls, or the seas of ice? Nothing would be possible if evolution hadn’t endowed us with eyes.

We recognize things because we have learned the shape of objects, we have learned to estimate that different shape from those we have encountered can be associated with the same object. We have learned by experience and because we were given the names of said objects. Like a supervised algorithm that needs a label to associate the shape, details, colors with a category. A dog and a wolf are very similar just across the pixels. Computer vision methods have enabled machines to be able to decipher these shapes and “learn” to classify them.

Now, algorithms, just like our eyes can identify in pictures or films, objects, or shapes. The methods are constantly evolving and perfecting to the point of reaching the so-called human level. But, there are several methods, image classification, object detection or recognition, and image segmentation. In this article, we will explore the image classification problem. The first part will present training a model from scratch, the second will present training with data augmentation, and the last transfer learning with pre-trained models.

Methods

Image Classification from scratch

Image classification can, when the volume of data you have is large enough, be done “from scratch”. The idea is to create a model and train it from scratch.

Like any classification problem, the data must be annotated. How to proceed when it comes to images? It’s quite simple in fact, the data of the same class must be stored in the same folder. It is necessary to take a folder per class or category considered. Like that:

> train/

... forest/

... img_1.jpeg

... img_2.jpeg

...

... moutain/

... img_1.jpeg

... img_2.jpeg

...

... sea/

... img_1.jpeg

... img_2.jpeg

...

validation/

... forest/

... img_1.jpeg

... img_2.jpeg

...

... moutain/

... img_1.jpeg

... img_2.jpeg

...

... sea/

... img_1.jpeg

... img_2.jpeg

test/

... forest/

... img_1.jpeg

... img_2.jpeg

...

... moutain/

... img_1.jpeg

... img_2.jpeg

...

... sea/

... img_1.jpeg

... img_2.jpeg

This simple way allows the model to associate a label with a picture.

After that, you build your neural network. From now, the standard is for Convolutional Neural Networks (CNN) when working with pictures. So you are going to build a CNN and train it with the INTEL data set. You’ll add a convolutional layer then a pooling layer, maybe a dropout layer to decrease the risk of overfitting and finishing with dense fully connected layers. The last one will output the results, or the prediction, The number of unit in this last layer is the number of classes you want to predict.

# building the model

model = tf.keras.Sequential([

layers.Conv2D(32, 3, activation='relu'),

layers.MaxPooling2D(),

layers.Dropout(0.5),

layers.Conv2D(32, 3, activation='relu'),

layers.MaxPooling2D(),

layers.Dropout(0.3),

layers.Conv2D(32, 3, activation='relu'),

layers.MaxPooling2D(),

layers.Flatten(),

layers.Dense(128, activation='relu'),

layers.Dense(num_classes, activation='softmax')

]) # compile the model with adam optimizer and sparse cross entropy model.compile(

optimizer='adam',

loss=tf.losses.SparseCategoricalCrossentropy(from_logits=True),

metrics=['accuracy']) # use early stopping to break the learning process if the model stop learning during 3 epochs

es = tf.keras.callbacks.EarlyStopping(patience=3)

history = model.fit(

train_x,train_y,

validation_data=(valid_x, valid_y), callbacks=[es], batch_size=32,

epochs=30

)

Here, I present a small CNN architecture where num_classes is the number of classes. In the context of this article we will predict 6 classes, so num_classes=6 . EarlyStopping constrains the model to stop when it overfits, the parameter patience=3 means that if during 3 epochs the model doesn’t improve, the training process is stopped.

If you have enough data and if your CNN is not too deep - but enough - to generate a good data representation you’ll obtain good results.

Unfortunately, is rarely the case and you need to try other options.

Data augmentation

So, if your model can’t obtain a good performance. You can change the architecture of your network. You can add or delete hidden layers. You can decrease or increase the number of units per layer. You can change the activation function or loss function. Or, you can change the preprocessing or your data.

But, what to do if it’s not enough?

You can use data augmentation. This technique permits you to create artificially (synthetic) images from yours in-memory (your original data will not be affected by this method). It consists of operations like rotation, the same picture will be rotated by different angles (creation of new images). Shifted, that is to say, that the pattern of the image will be offset from the frame, thus creating a “hole” which will have to be interpolated. This operation can be done horizontally or vertically. Zooming, the new image will be a zoom of a part in the original data, etc…

The perfect tool to do this is the object named ImageDataGenerator provided by Keras[2] ( keras.preprocessing.image.ImageDataGenerator() ).

from tensorflow.keras.preprocessing.image import ImageDataGenerator

data_gen = ImageDataGenerator(

rotation_range=40,

width_shift_range=0.2,

height_shift_range=0.2,

rescale=1./255,

shear_range=0.2,

zoom_range=0.2,

horizontal_flip=True,

fill_mode='nearest')

test_gen = ImageDataGenerator(rescale=1./255)

Here are just example values I used in the notebook.

This tool will create synthetic images to increase the volume of your dataset. How to use it?

Quick implementation of data augmentation used in a CNN. The results will be displayed in the Results section.

Unfortunately, yes again, you can have too few images to obtain good results. If your dataset is very small, even data augmentation can not save you. What do you do next?

Transfer Learning[3]

No, this is not the time to run away afraid of transfer learning. What is transfer learning? It is simply a method where you’ll use the knowledge learned for one task and exporting it to another.

In our case, transfer learning takes place with fairly large models (with millions or even hundreds of millions of parameters) that have been trained on a gigantic volume of data (the Imagenet[4] dataset) to generalize.

When you have a small dataset, the models you build cannot have a good representation of the data. You must therefore use pre-trained models that you will train on your data.

The method is simple, take the pre-trained model(s), freeze the weights of their layers, leave only the last layer, or the last layers to then train them with your data.

Neural networks specialize more and more according to their depth. The first layers will detect general patterns, lines for example. Then shapes will appear until you reach very fine details in the last layers. These are the ones that must be used to “tune” the model to your data. So rather than retraining the complete model with the ImageNet dataset plus your data (which would take months and require a significant investment of money) you can in minutes/hours obtain an extremely performing model using transfer learning.

In the notebook, I compare different pre-trained models to see which is the best for our study. To change the pre-trained model easily and quickly, the function below contains the architecture for tuning a pre-trained model on the data and evaluate it with metrics. The return is a data frame containing the results of the metrics, and the history of the model to plot the learning curves.

The next gist will show you how to use the function.

Yes, you need metrics to evaluate the performance of your different algorithms and you need to plot the learning curves (accuracy and loss) to look at the behavior of your training.

Metrics

To evaluate classification models different metrics can be used such as accuracy, precision, recall, f1-score, etc. (details of these metrics can be found here). The code below shows how to set up a metrics dictionary and the function that will be used to evaluate neural networks.

This function can be used in binary and multiclass classification problem.

Plotting learning curves

It’s very important when training a deep learning model to see the behavior of the learning curves to determine if the model as bias, overfit, or normal behavior. To do it, let’s see the code below which permits to plot the accuracy and loss curves for the training set and the evaluation set.

At this point, you know the different methods and the metrics used to evaluate the models.

Data

To avoid the classic MNIST[5] or FashionMNIST[6] for classification, we will take the dataset provided by INTEL (available on Kaggle[1]). These data are more fancies, they represent scenes from all over the world and represented 6 classes (buildings, forest, mountain, glacier, sea, and street). The data volume is also accessible for a project on a local computer because the training set is made up of 14k images, the validation set contains 3k images and 7k for the test. Each image has a shape of (150x150) pixels.