Preventing Fatalities by playing Video Games (Part 2)
By: Enrico Busto
Tags: Pixel Level Adaptation, Deep Learning, Domain Adaptation, Autonomous Vehicles, Autonomous Driving

In the previous article, I presented some challenges in training self-driving systems and two methods to overcome them: Pixel-level Adaptation and Features-level Adaptation, both based on Generative Adversarial Network. This article will explain how they work and show our experimental results.

Pixel-level Adaptation

The first method is based on a particular architecture of GAN called Cycle-Gan. This model can capture the principal style features from a set of images and then apply them to another image collection from another domain. For example, you can provide the model with two groups of images. The first one contains photos of horses and the second one of zebras. Then you can train the model to recolor the horse’s body with the zebra stripes.

This cycle-gan architecture is composed of 4 sub-modules: two Generators and two Discriminator. The training process is divided into two iterative phases: In the first phase, we train Discriminator models, and in the second phase, the Generator models.

A discriminator model is trained to recognize if a provided image is from the set of pictures from which it is trained on. We train two discriminator models, one for each of the two sets of photos.

Keeping the previous zebra/horse example, the first discriminator must indicate whether a photo contains a horse. The other discriminator, on the other hand, if it contains zebra. The discriminator model is a simple convolutional neural network (CNN).

Instead, both the generators have the objective of producing a new version of the given input image that must be realistic enough to fool the opponent discriminator. The generators are implemented as a Variational Autoencoder (VAE).

Now that we have defined the generator architecture, we need to define its behavior. We have two principal objectives. The first one is to fool the discriminator. To achieve this result, we use a loss formula called Adversarial Loss:

Reproducing the other domain’s style is not enough. The second Generator’s objective is to maintain the original image’s content.

When the first generator receives an image input and produces another image as output, this new image will be used as input from the second generator, which recreates the first generator's original input. We obtain this result by adding another loss called Cycle Loss.

The intuition behind this loss is that it forces the generator to create a new image that is realistic enough to fool the discriminator while maintaining enough information from the original input to allow the second generator to recreate the initial input.

First Experiment

We have created a new dataset consisting of both real and synthetic images. The real images are from the Cityscapes dataset, while the others are from a video game called GTA V.

We then used the cycle-gan to apply the real image style to the synthetic ones.

For our experiment, we trained a DeepLab V2 model on the GTA V dataset, and we tested on the Cityscape test set. We called this experiment “Source,” and we obtained a score of 37,96 MIoU.

Then we retrained the same network over our adapted version of the GTA V dataset, obtaining a score of 45.57% MIoU. We called this second test, “Adapted”.

To quantify the improvement, we made a final test. We have retrained the model directly with the cityscape dataset, and we tested on the cityscape test set. We obtained a result of 64,78% MIoU.

Using the Cycle-gan for the domain adaptation, we obtained a boost of about 20% classification performance with respect to what we achieved using just data from the original domain.

Check my next article: Preventing Fatalities by playing Video Games (Part 3), to see the results we got from our experiments on Feature-Level Adaptation