A Guide to Generative Adversarial Networks (GANs)
By: Danyal Jamil
Tags: Resnet, Deep Learning, Nanodegree, Gans, Neural Networks

A Guide to Generative Adversarial Networks (GANs)

Day- 18

Hello everyone! This is my fourteenth writing on my journey of Completing the Deep Learning Nanodegree in a month! In this writing, we’ll discuss GANs.

Today was a very awesome day because I started and finished the module, ‘Generative Adversarial Networks’ in just one day! GANs are so interesting that I couldn’t leave studying about them. Let’s get into it.

Generative Adversarial Networks

Generative Adversarial Networks belong to the set of generative models. It means that they are able to produce / to generate new content.

We had AutoRegressive Models which used to generate one pixel at a time, but in GANs, we have a Generator Network, which takes in a noisy image, then runs that noise through a differentiable function and reshapes it to have recognizable structure. Running this network for many noisy images can result into many different outputted images. Training GANs and Supervised Models is very different. In supervised Models, we have to input the pictures and their labels into the model in order for it to learn but in this case, we just pass in the images.

GANs have another part in them, Discriminator, which guides the Generator, it basically outputs the probability that the image from the Generator is real. Hence, the Generator is trying its best to get 1 as the output from the Discriminator.

The idea behind GANs is that you have two networks, a generator G and a discriminator D, competing against each other. The generator makes “fake” data to pass to the discriminator. The discriminator also sees real training data and predicts if the data it’s received is real or fake. The word ‘adversarial’ in GANs means that the two parts, Generator and Discriminator are in a competition with each other.

The cost for the Discriminator is the negative of that of Generator.

Training

For training GANs, we have value function. The Generator wants to minimize the Value function and the Discriminator wants to maximize it. We reach at equilibrium when both have achieved their aims, and nothing can be done to improve any one’s approach. This point is called the Saddle Point.

Saddle Point

In mathematics, a saddle point is a point on the surface of the graph of a function where the slopes, derivatives, in orthogonal directions are all zeros, but which is not a local extremum of the function.

But unfortunately, even though the Equilibrium exists, we may not be able to find it.

Saddle Point

Working

We’ll run two optimizations simultaneously, one for the Generator and the other for the Discriminator, both with respect to their own parameters. But, optimization functions do not necessarily find the equilibrium in this case. The problem can be in the case of our having clusters of images to input and then the Generator learns to make one cluster, the Discriminator rejects it and then the Generator makes the second and the Discrimnator rejects that too and so on..

The most important thing in GANs is to make a good architecture. In any architecture, we need Hidden layers in both, Generator and the Discriminator. And the best activation functions are Leaky ReLUs. In a Generator, we use TahH as the last function and Sigmoid for the Discriminator, to get the probability out of it.

Leaky ReLUs

We should use a leaky ReLU to allow gradients to flow backwards through the layer unimpeded. A leaky ReLU is like a normal ReLU, except that there is a small non-zero output for negative input values.

Leaky ReLU

We define two losses.

Then, we define two optimizers, both separate for both losses.

Then we define the loss function, commonly used is BCE, we multiply the labels with a number ever so slightly lesser than 1 to normalize it and to avoid any large steps.

d_loss = nn. BCEWithLogitsLoss(logits, labels*0.9)

Then, we make another loss function for the Generator, with the labels flipped.

g_loss = nn. BCEWithLogitsLoss(logits, flipped_labels)

We will ultimately use BCEWithLogitsLoss, which combines a sigmoid activation function and and binary cross entropy loss in one function.

Here are sample code snippets for an MLP Generator and Discriminator codes.

#Discriminator class

class Discriminator(nn.Module): def __init__(self, input_size, hidden_dim, output_size):

super(Discriminator, self).__init__()



# define all layers

self.sig = nn.Sigmoid()

self.hid = nn.Linear(input_size, hidden_dim*4)

self.hid1 = nn.Linear(hidden_dim*4, hidden_dim*2)

self.hid2 = nn.Linear(hidden_dim*2, output_size)

self.drop = nn.Dropout(0.2)



def forward(self, x):

# flatten image

x = x.view(-1, 28*28)

# pass x through all layers

x = self.hid(x)

x = F.leaky_relu(x, 0.2)

x = self.drop(x)

x = self.hid1(x)

x = F.leaky_relu(x, 0.2)

x = self.drop(x)

x = self.hid2(x) return x <------------------------------------------------------------------> #Generator class

class Generator(nn.Module): def __init__(self, input_size, hidden_dim, output_size):

super(Generator, self).__init__()



# define all layers

self.tanh = nn.Tanh()

self.hid = nn.Linear(input_size, hidden_dim*2)

self.hid1 = nn.Linear(hidden_dim*2, hidden_dim*4)

self.hid2 = nn.Linear(hidden_dim*4, output_size)

self.drop = nn.Dropout(0.2)



def forward(self, x):

# flatten image

# pass x through all layers

x = self.hid(x)

x = F.leaky_relu(x, 0.2)

x = self.drop(x)

x = self.hid1(x)

x = F.leaky_relu(x, 0.2)

x = self.drop(x)



# final layer should have tanh applied

x = self.tanh(self.hid2(x)) return x

But this setup is for small images and we use Feed Forward NNs in these architectures. But for huge images, we use Convolutional NNs.

But normally we use pooling layers to decrease the size of the images, but here, we, certainly, want to do the opposite. We take in a small image and increase its size.

Deep Convolutional GAN (DCGANs)

Convolutional GAN

Discriminator

In this special Discriminator, we use CNNs instead of MLPs. And we do not have maxpooling layers in between the Convolutional Layers and we decrease the size of the image by using stride=2.

What setting stride to 2 does? Suppose we input a 32x32 image, the image in the Convolutional Layer will be 16x16, yes, half of the dimension of the image.

We also perform Batch Normalization on all the layers. And what it does is sets the Mean to 0, and Variance to 1. This step helps the model to train faster and covers the problems caused because of poor hyperparameter settings.

Then, at the final Convolutional Layer, we flatten the image and pass it through a fully connected layer. And then Sigmoid at the last.

Generator

Here, we use the Convolutional Encoders and Decoders.

Encoders to decrease the spacial information of the image to make the training easier and then decoder to get the spacial size back.

First step is to pass the ‘z’ through a fully connected layer and then into a transpose convolutional layer and from there, we upsample the image until the size requirement is matched. We use Batch Normalization and ReLU Function on all the hidden layers and TanH activation function on the last layer.

It basically upsamples a vector into a sinlge layer image.

Batch Normalization is another part of this.

The idea is that, instead of just normalizing the inputs to the network, we normalize the inputs to every layer within the network.

For Scaling, we use the following easy code.

def scale(image, feature_range=(-1, 1)):

min, max = feature_range

image = image * (max - min) + min

return x

And for total loss,

Total Loss

Image to Image Translation

We know that we had been dealing with Loss Functions to measure the correctness of the model and is the basic parameter which a Model Trainer keeps in view to find out when he’s done the job.

An objective function is typically a loss function that you seek to minimize, or in some cases maximize, during training a neural network. These are often expressed as a function that measures the difference between a prediction y_hat and a true target y.

But in this case, how can we define a Loss Function for images, how can we make a function that calculates the difference between two images, the generated and that from a true dataset. We can consider Euclidean Distance formula here, but that’s way too simple to be worked with.

Latent Space

Latent means “hidden” or “concealed”. In the context of neural networks, a latent space often means a feature space, and a latent vector is just a compressed, feature-level representation of an image!

For example, when you created a simple auto encoder, the outputs that connected the encoder and decoder portion of a network made up a compressed representation that could also be referred to as a latent vector.

We’ll now discuss two applications of GANs.

Pix2Pix

What this model is that it takes in an image of an incomplete drawing and then shows the best resemblance of it according to its knowledge.

Now, the Pix2Pix Generator needs to input an image and then it needs to work its way down. Hence, we’ll have an Encoder and Decoder in the Generator function that will take in an Image, the Encoder will have CNNs, convert it to feature level representation and then pass it in the Decoder and then generate the image.

Now, the problem arises with the Discriminator, that if we pass an image of a real bat, it’ll classify it real. But if our drawing is of a flower and the discriminator received the same image of the bat, it’ll classify it as real. And here is the problem, despite the fact that the bat image has nothing to do with the initially inputted flower drawing, the Discriminator will classify it as real. Hence, we change the Discriminator to take in a pair of both the images, the drawing and the generated and to find if the pair is real or not.

And this is how a Pix2Pix Model works, but what if you don’t have the data to train on, what do we compare the generated image on that basis. Let’s discuss the model of CycleGANs here.

Pix2Pix example

CycleGAN

How Pix2Pix works is that is has a mapping function Gx -> y. Here, we have the X and Y, unpaired and we need to come up with a way to make the ‘G’ mapping correct in order to correctly pair the two unpaired set of images.

But for this kind of models, only one mapping is not enough. We also need to consider an opposite, inverse mapping. We have this to check if the mapping done by ‘G’ is correct or till what measure it is, Gy -> x.

Let’s understand these mapping by using an example, suppose we pass in an English sentence to be translated into German. Then the Inverse Mapping must take in the German Translated text and output the same text as that was initially inputted. Similarly, if we pass an image of a horse and want it to become a picture of a zebra, then the generated image of the zebra must get the original image of the horse once put through the Inverse Mapping.

CycleGAN is made of two discriminator and two generator networks.

CycleGAN example

Resnet block

What exactly is a Resnet block? It may sound familiar from something like ResNet50 for image classification, pictured below.

ResNet50

ResNet blocks rely on connecting the output of one layer to the input of an earlier layer. The motivation for this structure is as follows: very deep neural networks can be difficult to train. Deeper networks are more likely to have vanished or exploding gradients and, therefore, have trouble reaching convergence; batch normalization helps with this a bit. However, during training, we often see that deep networks respond with a kind of training degradation. Essentially, the training accuracy stops improving and gets saturated at some point during training. In the worst cases, deep models would see their training accuracy actually worsen over time!

One solution to this problem is to use Resnet blocks that allow us to learn so-called residual functions as they are applied to layer inputs.

The connection that residual blocks make sometimes calls a slip connection. By summing up the output of one layer to the input of a previous layer, we are effectively making a connection between layers that are not in sequence; we are skipping over at least one layer with such a connection, as is indicated by the loop arrow below.