Conquer Class Imbalanced Dataset Issues using GANs | by Swanand Mhalagi | Towards Data Science

A common problem in real life deep learning applications is that some classes have a significantly higher number of examples in the training set than other classes. Let’s consider a manufacturer is building a visual inspection system to detect damaged products. No manufacturer will have as many damaged product images as good product images. This difference is referred to as class imbalance. Class imbalanced datasets are common across different domains such as health, banking, security and others. With such datasets, the learning algorithms are often biased towards the majority classes and therefore there is a higher miss-classification rate for the minority class instances.
There are different strategies such as oversampling, under sampling, two phase training and cost sensitive learning to address this problem. Methods that generate artificial data for the minority class constitute a more general approach compared to algorithmic improvements. This article is all about reducing this gap in datasets using Deep Convolution Generative Adversarial Networks (DC-GAN) to improve classification performance.
GANs are like Rubik's cube. Once you know the trick to solve the cube, there are multiple ways to get to the perfect cube. It can be as fast as 3.47 sec (current world record for 3x3 Rubik’s cube) and it's easy to discover if you have messed up. But if you don’t know the trick to solve the cube it can take forever. The time & cost to failure for GAN is pretty high (considering training time and dollar value of resources), especially when you have limited resources and you might still wonder what went wrong. Once you know GAN dependencies, it will be relatively easy to get to a perfect solution.
Throughout this article we will tackle the following topics -
Tips and tricks to make GANs work/stable.How to define GAN. How to shape GAN for real world datasets of Kaggle competitions ?Exciting use cases of GANs.
Tips and tricks to make GANs work/stable.
How to define GAN. How to shape GAN for real world datasets of Kaggle competitions ?
Exciting use cases of GANs.
Tips and tricks to make GANs work/stable -
Developing a GAN for generating images requires a discriminator convolutional neural network model for classifying whether a given image is real or generated and a generator model that uses inverse convolutional layers to transform an input to a full two-dimensional image of pixel values.
These generator and discriminator models compete in a zero-sum game. It means that improvements to one model come at the cost of degrading of performance in the other model. The result is a very unstable training process that can often lead to failure.
While research in GANs continues to improve fundamental stability of these models, there are bunch of tricks to train them and make them stable.
Use Strided Convolution >> Do not use max pooling layers and instead use the stride in convolutional layer to perform down sampling in the discriminator model. Use Conv2DTranspose and stride for up sampling.Remove Fully-Connected Layers >> Fully-Connected layers are not used in discriminator, instead convolutional layers are flattened and passed directly to the output layer.Use Batch Normalization >> Batch norm layers are recommended in both the discriminator and generator models, except the output of the generator and input to the discriminator.Use of ReLU, Leaky ReLU and Tanh >> ReLU is recommended only for generators but for discriminator variation of ReLU that allows values less than zero, called Leaky ReLU is preferred. Additionally, the generator uses Tanh and discriminator uses Sigmoid activation function in the output layer.Normalize the input >> Normalize the input images between -1 to 1. Construct different mini-batches for real and fake, i.e. each mini-batch needs to contain only all real images or all generated images.Learning rates >> Use different learning rates for discriminator (1e-3) and generator (1e-4). Use Adam optimizer for both.Performance hack >> Train discriminator twice and generator once. Use 50% dropout in generator.Track failures early >> A loss of 0.0 in the discriminator is a failure mode. If the loss of the generator steadily decreases, it is likely fooling the discriminator with garbage images. When training is working smooth, discriminator loss has low variance and goes down overtime.
Use Strided Convolution >> Do not use max pooling layers and instead use the stride in convolutional layer to perform down sampling in the discriminator model. Use Conv2DTranspose and stride for up sampling.
Remove Fully-Connected Layers >> Fully-Connected layers are not used in discriminator, instead convolutional layers are flattened and passed directly to the output layer.
Use Batch Normalization >> Batch norm layers are recommended in both the discriminator and generator models, except the output of the generator and input to the discriminator.
Use of ReLU, Leaky ReLU and Tanh >> ReLU is recommended only for generators but for discriminator variation of ReLU that allows values less than zero, called Leaky ReLU is preferred. Additionally, the generator uses Tanh and discriminator uses Sigmoid activation function in the output layer.
Normalize the input >> Normalize the input images between -1 to 1. Construct different mini-batches for real and fake, i.e. each mini-batch needs to contain only all real images or all generated images.
Learning rates >> Use different learning rates for discriminator (1e-3) and generator (1e-4). Use Adam optimizer for both.
Performance hack >> Train discriminator twice and generator once. Use 50% dropout in generator.
Track failures early >> A loss of 0.0 in the discriminator is a failure mode. If the loss of the generator steadily decreases, it is likely fooling the discriminator with garbage images. When training is working smooth, discriminator loss has low variance and goes down overtime.
All such tricks we need to put together to create a perfect GAN.
How to define GAN? How to shape GAN for real world datasets of Kaggle competitions? -
Researchers have developed multiple flavor GANs and there are multiple solutions out there on web to address class imbalance issue but all of these are mainly on toy datasets such as MNIST, CIFAR-10 and ImageNet. I have been working on a medical image classification (Diabetic Retinopathy Detection) dataset from Kaggle competitions. This dataset has 4 classes where class 1 has 13k samples whereas class 4 has only 600. One of the recent methodology used by Kaggle competition winners to address class imbalance issue is nothing but use of DC-GAN. We will use DC-GAN to create artificial samples for class 4 of Diabetic Retinopathy Detection database.
Let's begin the coding exercise and implement these tricks to generate new images. All the experiments are done on NVIDIA Quadro M400 GPU. This GPU has a 8 GB RAM. GPU is configured with CUDA 9.0, cuDNN 7.4, TF for GPU 1.8.0 on Ubuntu 16.04 instance.
Import all the necessary environments. This code expects all the packages are installed beforehand.
I have already downloaded the database from Kaggle. The images in this database are hard to distinguish by naked eyes hence this dataset is very different from all toy datasets. The database has several zip files, we need to unzip them into train/test folders containing respective images. All the labels for train images are provided in a separate csv file.
In the code below we will read a csv file containing labels and name of the images. We need to do some sanity checks (adding .jpeg extension, delete all the images of size 0 KB, remove entries of deleted images from the data frame) before we move forward.
In this dataset class 3 and 4 are minority classes since they have very low representation in entire dataset. We will train GAN to generate images for class 4.
Below section defines discriminator and generator. The discriminator uses convolution layer with 2 x 2 strides to down sample the input image (Trick #1 & 2). The output layer use Sigmoid activation function to predict whether the input sample is real or fake. The model is trained to minimize binary cross entropy loss function with the use of Adam optimizer (Trick #4).
The generator is defined with Conv2DTranspose with 2 x 2 strides to up-sample the images up-to 128 pixels. The output layer uses Tanh activation function to ensure output values are in desired range of [-1, 1] (Trick #4). We have intentionally used different learning rates for discriminator and generator (Trick #6).
The define_gan() function uses already defined generator and discriminator models and create a new logical model.
The function load_real_samples() below will read actual data from the dataset and normalize the images (Trick #5) before feeding them to generate_real_samples() while invoking train().
The generate_fake_samples() function generates images with random pixel values and a fake label 0.
There are no metrics to evaluate GAN performance, the images generated must be evaluated for quality by human operator. This makes hard to decide where to stop the training.
Due to adversarial nature, the generator properties are changing after every epoch. Once an acceptable quality of image is generated, the generator may not improve the performance, in many cases even degrade with subsequent epochs.
This problem is handled using 3 options
Periodically evaluate the classification accuracy of discriminator on real and fake images.Periodically generate images and save them for human operator review.Periodically save the generator model for later use.
Periodically evaluate the classification accuracy of discriminator on real and fake images.
Periodically generate images and save them for human operator review.
Periodically save the generator model for later use.
All this action will be performed by summarize_performance() function to evaluate discriminator model. Training GAN for multiple epochs will generate snapshots of model every 10th epoch simultaneously save_plat() will keep saving images. This will be helpful to back track the progress of GAN image generation.
The train() function is defined below, the number of batches within each epoch is defined by how many times the batch size divides into the training dataset.
Train discriminator model is updated twice (with fake and real samples each time) and generator once for each batch iteration (Trick #7).
It is critical to observe the loss while training the GAN, sudden drop in discriminator loss indicates that the generator model has started generating bad samples that discriminator can easily discriminate (Trick #8).
In order to visualize the models we can plot the them using plot_model() function.
The summary() function can also be used to see the model layout and number of trainable parameters. Call the train() function to begin the discriminator and generator training.
Discriminator model
_________________________________________________________________Layer (type)                 Output Shape              Param #   =================================================================conv2d_1 (Conv2D)            (None, 128, 128, 16)      448       _________________________________________________________________leaky_re_lu_1 (LeakyReLU)    (None, 128, 128, 16)      0         _________________________________________________________________conv2d_2 (Conv2D)            (None, 64, 64, 8)         1160      _________________________________________________________________leaky_re_lu_2 (LeakyReLU)    (None, 64, 64, 8)         0         _________________________________________________________________conv2d_3 (Conv2D)            (None, 32, 32, 16)        1168      _________________________________________________________________leaky_re_lu_3 (LeakyReLU)    (None, 32, 32, 16)        0         _________________________________________________________________conv2d_4 (Conv2D)            (None, 16, 16, 8)         1160      _________________________________________________________________leaky_re_lu_4 (LeakyReLU)    (None, 16, 16, 8)         0         _________________________________________________________________flatten_1 (Flatten)          (None, 2048)              0         _________________________________________________________________dropout_1 (Dropout)          (None, 2048)              0         _________________________________________________________________dense_1 (Dense)              (None, 1)                 2049      =================================================================Total params: 11,970Trainable params: 5,985Non-trainable params: 5,985_________________________________________________________________
Generator model
_________________________________________________________________Layer (type)                 Output Shape              Param #   =================================================================dense_2 (Dense)              (None, 65536)             6619136   _________________________________________________________________leaky_re_lu_5 (LeakyReLU)    (None, 65536)             0         _________________________________________________________________reshape_1 (Reshape)          (None, 16, 16, 256)       0         _________________________________________________________________conv2d_transpose_1 (Conv2DTr (None, 32, 32, 128)       524416    _________________________________________________________________leaky_re_lu_6 (LeakyReLU)    (None, 32, 32, 128)       0         _________________________________________________________________conv2d_transpose_2 (Conv2DTr (None, 64, 64, 128)       262272    _________________________________________________________________leaky_re_lu_7 (LeakyReLU)    (None, 64, 64, 128)       0         _________________________________________________________________conv2d_transpose_3 (Conv2DTr (None, 128, 128, 128)     262272    _________________________________________________________________leaky_re_lu_8 (LeakyReLU)    (None, 128, 128, 128)     0         _________________________________________________________________conv2d_5 (Conv2D)            (None, 128, 128, 3)       3459      =================================================================Total params: 7,671,555Trainable params: 7,671,555Non-trainable params: 0_________________________________________________________________
The training processes will look like below snippet
.....>319, 5/10, d1=0.692, d2=0.761 g=0.709>319, 6/10, d1=0.822, d2=0.759 g=0.690>319, 7/10, d1=0.733, d2=0.764 g=0.723>319, 8/10, d1=0.662, d2=0.740 g=0.743>319, 9/10, d1=0.701, d2=0.683 g=0.758>319, 10/10, d1=0.830, d2=0.744 g=0.728>320, 1/10, d1=0.749, d2=0.717 g=0.731>320, 2/10, d1=0.677, d2=0.796 g=0.722>320, 3/10, d1=0.766, d2=0.700 g=0.717>320, 4/10, d1=0.676, d2=0.736 g=0.765>320, 5/10, d1=0.792, d2=0.762 g=0.730>320, 6/10, d1=0.690, d2=0.710 g=0.719>320, 7/10, d1=0.807, d2=0.759 g=0.708>320, 8/10, d1=0.715, d2=0.747 g=0.711>320, 9/10, d1=0.719, d2=0.720 g=0.731>320, 10/10, d1=0.695, d2=0.717 g=0.694################# Summarize ###################>Accuracy real: 35%, fake: 57%
After 320 epochs, below is the quality of images I was able to produce. Training for this GAN took ~ 30 minutes. More complex generator and discriminator model could have produced better quality images.
Progression of the GAN towards a perfect cube
Now these newly generated minority class images can be added to the original imbalanced dataset. This will help convert imbalanced multi class data into a balanced dataset. This will improve the classification performance of the model.
An image classification algorithm for this imbalanced dataset can be found here
towardsdatascience.com
All the iPython notebooks are available at https://github.com/swanandM/DC-GAN-RetinopathyImages
Image classification notebooks https://github.com/swanandM/Diabetic-Retinopathy-Detection-with-TF
Exciting use cases of GANs
GAN has made impressive progress in its first few years. In 2017, GAN was able to create a 1024 x 1024 size images which were able to fool almost everyone. Although this kind of technology can create serious social trustworthiness issue considering the amount of fake news getting publish these days. Engineers have to be careful before developing any such application that will generate social unrest.
Creating animation for games as well as for movies takes tremendous efforts. GAN can auto-generate these animation characters without any professional skills. Besides this, a professional animation creator can take inspiration from these designs.
CycleGAN can transform images from one domain to another domain.
PixelDTGAN is used to generate marketing images based on celebrity pictures. This is also used as a suggestion tool based on images.
Text to image is one of the application of domain transfer GAN. This has tremendous applications including photo editing and computer aided design.
Creating emoji from pictures is one of the well-known application already been used in platforms such as Snap chat and Instagram.
References
Effective data generation for imbalanced learning using conditional generative adversarial networksHow to Develop a GAN to Generate CIFAR10 Small Color PhotographsSome cool applications of GANHow to train a GAN?
Effective data generation for imbalanced learning using conditional generative adversarial networks
How to Develop a GAN to Generate CIFAR10 Small Color Photographs
Some cool applications of GAN
How to train a GAN?