DCGANs — Generating Dog Images with Tensorflow and Keras | by Konstantin Georgiev | Towards Data Science

This post is a tutorial on the basic ideas behind the effectiveness of DCGANs, as well as some methods/hacks to improve their performance. These methods are all based on my experience during Kaggle’s Generative Dogs Competition. The tutorial is also available, either in notebook format on my original kernel, or on GitHub.
To run this example locally or using Colab, you will need a Kaggle account, in order to retrieve its API key and use the provided datasets. Full tutorial is available here.
Unlike most popular neural network architectures, GANs are trained to solve two problems simultaneously — discrimination (effectively separating real from fake images) and “realistic” fake data generation (effectively generating samples considered as real). As we can see these tasks are complete polar opposites but what would happen if we separated them into different models?
Well, the general names for these models are Generator (G) and Discriminator (D) and are considered the building blocks behind the theory of GANs.
The Generator network takes as input a simple random noise N-dimensional vector and transforms it according to a learned target distribution. Its output is also N-dimensional. The Discriminator on the other hand models a probability distribution function (like a classifier) and outputs a probability that the input image is real or fake [0, 1]. With this in mind, we can define the two main goals of the generation task:
1. Train G to maximise D’s final classification error. (So that the generated images are perceived as real).
2. Train D to minimise the final classification error. (So that real data is correctly distinguished from fake data).
To achieve this, during backpropagation, G’s weights will be updated using Gradient Ascent to maximise the error, while D will use Gradient Descent to minimise it.
So how do we define a loss function that estimates the cumulative performance of the two networks? Well, we can use the Absolute Error to estimate D’s error and then we can reuse the same function for G but maximised:
In this case, p_t represents the true distribution of images, while p_g is the distribution created from G.
We can observe that this theory is based on some key concepts of Reinforcement Learning. It can thought of as a two-player minimax game, where the two players are competing against each other and thus progressively improving in their respective tasks.
We saw the basic idea behind the theory of GANs. Now let’s take one step further and learn how DCGANs work by applying ideas and methods from Convolutional Neural Networks.
DCGANs utilize some of the basic principles of CNNs and have thus become one of the most widely used architectures in practice, due to their fast convergence and also due to the fact that they can be very easily adapted into more complex variants (using labels as conditions, applying residual blocks and so on). Here are some of the more important problems that DCGANs solve:
D is created so that it basically solves a supervised image classification task. (for this case dog or no dog)
The filters learned by the GAN can be utilized to draw specific objects in the generated image.
G contains vectorized properties that can learn very complex semantic representations of objects.
Here are some of the core guidelines to consider when creating a stable DCGAN, as opposed to a standard CNN(taken from the official paper):
Replace Pooling functions with Strided convolutions. (this allows D to learn its own spatial downsampling and G its respective upsampling without adding any bias to the model)
Use BatchNorm (it stabilizes learning by normalizing the input to each unit to have zero mean and unit variance, this also helps to create more robust deep models without having the gradients diverging)
Avoid using Fully-Connected hidden layers (not output). (example for this is global average pooling which seems to hurt convergence speed)
For G — use ReLU activations and Tanh for the output. (tanh is generally the more preffered activation when you have an image as an output as it has a range of [-1, 1])
For D — use LeakyReLU activations (and a sigmoid function for the output probabilites). (this is tested empirically and seems to work well for modelling to a higher resolution)
Here is the most standard structure of a DCGAN generator:
As we can observe, its initial input is simply a (1, 100) noise vector, which passes through 4 Convolutional layers with upsampling and a stride of 2 to produce a result RGB image of size (64, 64, 3). To achieve this, the input vector is projected onto a 1024-dimensional output to match the input of the first Conv layer, which we will see more later on.
What would a standard discriminator look like? Well, about what you would expect, let’s take a look:
This time around we have an input image of (64, 64, 3), same as G’s output. We pass it to 4 standard downsampling Conv layers, again with a stride of 2. In the final output layer, the image gets flattened to a vector, which is usually fed to a sigmoid function, which then outputs D’s prediction for that image (a single value representing the probability in the range [0,1] — dog = 1 or no dog = 0).
Well, now you saw the basic idea behind GANs and DCGANs, so now we can proceed to generate some dogs using Tensorflow and Keras :).
Before we proceed with creating the GAN model let’s first do a quick exploration of the Stanford Dogs dataset, which we’ll be using. Because we also have the annotations for each image, we can use them to map each dog image to its respective breed. To do that, we can first make a dictionary that maps the breed code in the file name to the actual breed name.
Next, I’m going to use OpenCV for a quick function to read and transform the images to RGB.
If we look at the dataset, we can see that each annotation folder contains a list of xml files. These files are associated with a specific image and contain very useful information, mainly the bounding boxes around each dog in the image. There are also images with more than one dog in them and this allows us to accurately crop them and make a dataset that contains single dogs only images.
Here we can utilize the xml library to create a tree and find the relevant elements for that annotation. For each object we can extract the bounding box coordinates, crop the images and normalize the crop by shrinking or expanding it depending on the result image width. Finally, we’ll save the images in a numpy array.
for o in objects:                              bndbox = o.find('bndbox')                               xmin = int(bndbox.find('xmin').text)                              ymin = int(bndbox.find('ymin').text)                              xmax = int(bndbox.find('xmax').text)                              ymax = int(bndbox.find('ymax').text)                                 .. Add some margins and adjust the width ..                             # Crop the image                              img_cropped = img[ymin:ymin+w, xmin:xmin+w, :]      # [h,w,c]                           .. Interpolation step ..                           # Resize the image                              img_cropped = cv2.resize(img_cropped, (image_width, image_height), interpolation=interpolation)# Save the images and labelsdog_images_np[curIdx,:,:,:] = np.asarray(img_cropped)dog_breed_name = dog_breed_dict[dog_ann.split('_')[0]]breeds.append(dog_breed_name)curIdx += 1                return dog_images_np, breeds
The features take about 2–3 mins to load.
By trial, we can figure out that the result images with single dogs only are 22125 and thus we can specify the exact size of our numpy array. There are 120 different dog breeds.
Now that we have the features and labels, we can plot them in a square grid to see what the crops look like and to make sure that they are labeled correctly.
def plot_features(features, labels, image_width=image_width, image_height=image_height, image_channels=image_channels,examples=25, disp_labels=True):       if not math.sqrt(examples).is_integer():        print('Please select a valid number of examples.')        return    imgs = []    classes = []    for i in range(examples):        rnd_idx = np.random.randint(0, len(labels))        imgs.append(features[rnd_idx, :, :, :])        classes.append(labels[rnd_idx])    fig, axes = plt.subplots(round(math.sqrt(examples)), round(math.sqrt(examples)),figsize=(15,15),    subplot_kw = {'xticks':[], 'yticks':[]},    gridspec_kw = dict(hspace=0.3, wspace=0.01))        for i, ax in enumerate(axes.flat):        if disp_labels == True:            ax.title.set_text(classes[i])        ax.imshow(imgs[i])
Note that we need to normalize the pixel values to make sure that the dogs are plotted correctly.
plot_features(dog_images_np / 255., breeds, examples=25, disp_labels=True)
Here we have a whole list of hyperparameters that you can tune and play around with to try and improve the model. I’ve mostly gathered these values from research papers and experimented a bit with tweaking them and this is what I ended up with. Here’s a list of things you could try out:
Sample size — the number of features
Batch size — 64 or 32 could improve performance, but are computationally heavy and you can only run the model for a small amount of epochs
Weight Init Std and Mean — these values are come from research papers and seem to stabilize model training
Leaky ReLU slope — the threshold for D’s activations, also seems robust
Downsize factor and Scale factor — set up so that G’s noise vector can be reshaped to (4, 4, 512), other combinations might also work
Dropout — amount of dropout layers, their placement and their rate could improve performance.
Learning rate and Learning rate decay — very important to model convergence, hard to tune precisely, G and D can have different learning rates.
Noise vector shape — usually 128 or 100 seems to be sufficient
Now let’s use our numpy array of features to construct a Tensorflow dataset object. First, we can convert the data types to float32, which always helps to preserve some memory.
dog_features_tf = tf.cast(dog_images_np, 'float32')
We could also apply Data Augmentation to our dataset. This includes random horizontal flips, zooming and cropping the image in random regions. Out of these, I found only the first method to be somewhat useful for adding some more variance to the dataset, as the other methods introduce a lot of noise.
So in this case, there will be a 50% chance that an image in our dataset will be flipped from left to right.
def flip(x: tf.Tensor) -> (tf.Tensor):    x = tf.image.random_flip_left_right(x)    return x
Now we can use Tensorflow to create the dataset by shuffling it, applying some augmentation, and finally separating it into batches of the specified Batch size.
dog_features_data = tf.data.Dataset.from_tensor_slices(dog_features_tf).shuffle(sample_size).map(flip).batch(batch_size, drop_remainder=True)
Before we actually make the Generator, let’s see a few normalizations that can gradually speed up a DCGAN’s convergence.
One of these methods is Weight initialization. It turns out that it’s pretty important for training stable GANs. Firstly, the model weights need to be zero-centered with a slight increase of std(0.02). This stabilizes both D and G during training and prevents the model gradients from vanishing or exploding. This is a key step in every case, where we have to use random variables in our model (the random noise vector).
Here is an example of how Weight initialization can seriously affect the learning process of a Neural Network.
We can also apply a Truncated Normal distribution using Keras, which will discard values more than 2 standard deviations from the mean. This could perhaps eliminate some outlier points during training.
weight_initializer = tf.keras.initializers.TruncatedNormal(stddev=weight_init_std, mean=weight_init_mean, seed=42)
Spectral Normalization is a new type of weight initialization, designed specifically for GANs, which seems to further stabilize model training (you can read more from this paper). For a more detailed explanation on Spectral Normalization and why it works it also worth to check out this post, it has very intuitive examples.
Spectral Normalization of a single weight in our network can be defined as the following:
Here u and v are simple random vectors of the same size. They are utilized to perform what’s called a power iteration operation on the specific weight, for each learning step and it proves to be a lot more computationally efficient than simply penalizing the gradients.
Afterwards, in the backpropagation step, we use WSN(W) to update the weights instead of W.
For this project, I will reuse some custom Keras layers implemented by IShengFang (official code), to apply Spectral Normalization on top of Conv and Dense layers.
Here is also a good example on the effect of Spectral Normalization:
Let’s also define some template layers in Keras, so that we can create G and D more easily later on. The standard pattern for the layers will be:
TP_Conv_Block = [(Conv(SN)2DTranspose (upsample)) -> (BatchNorm) -> (ReLU)]
Conv_Block = [(Conv(SN)2D (downsample)) -> (BatchNorm) -> (LeakyReLU)]
For this example, I will be using standard Conv2DTranspose blocks in G and Spectral Normalized Conv2D layers in D.
def transposed_conv(model, out_channels, ksize, stride_size, ptype='same'):    model.add(Conv2DTranspose(out_channels, (ksize, ksize),                              strides=(stride_size, stride_size), padding=ptype,                               kernel_initializer=weight_initializer, use_bias=False))    model.add(BatchNormalization())    model.add(ReLU())    return modeldef convSN(model, out_channels, ksize, stride_size):    model.add(ConvSN2D(out_channels, (ksize, ksize), strides=(stride_size, stride_size), padding='same',                     kernel_initializer=weight_initializer, use_bias=False))    model.add(BatchNormalization())    model.add(LeakyReLU(alpha=leaky_relu_slope))    #model.add(Dropout(dropout_rate))    return model
We can finally define our generator. The model structure is mostly based on the official DCGAN paper with a few tweaks that I found beneficial to the performance. Here is the overall structure:
[Input(128, 1) -> Dense(2048,) -> Reshape(4, 4, 128) -> TP_Conv_Block(Depth=512, K=5x5, S=1x1) -> Dropout(0.5) -> TP_Conv_Block(Depth=256, K=5x5, S=2x2) -> Dropout(0.5) -> TP_Conv_Block(Depth=128, K=5x5, S=2x2) -> TP_Conv_Block(Depth=64, K=5x5, S=2x2) -> TP_Conv_Block(Depth=32, K=5x5, S=2x2) -> Dense(Depth=3, Tanh)]
def DogGenerator():    model = Sequential()    model.add(Dense(image_width // scale_factor * image_height // scale_factor * 128,                    input_shape=(noise_dim,), kernel_initializer=weight_initializer))    #model.add(BatchNormalization(epsilon=BN_EPSILON, momentum=BN_MOMENTUM))    #model.add(LeakyReLU(alpha=leaky_relu_slope))    model.add(Reshape((image_height // scale_factor, image_width // scale_factor, 128)))        model = transposed_conv(model, 512, ksize=5, stride_size=1)    model.add(Dropout(dropout_rate))    model = transposed_conv(model, 256, ksize=5, stride_size=2)    model.add(Dropout(dropout_rate))    model = transposed_conv(model, 128, ksize=5, stride_size=2)    model = transposed_conv(model, 64, ksize=5, stride_size=2)    model = transposed_conv(model, 32, ksize=5, stride_size=2)        model.add(Dense(3, activation='tanh', kernel_initializer=weight_initializer))    return model
The Discriminator is relatively easier to implement, since its basically a small CNN two-class classifier. We can choose whether to apply Spectral Normalization or not and see the performance effects. For this example, I will try to apply SN only in D. Here is D’s structure:
[Input(128, 128, 3) -> Conv(SN)2D(Depth=64, K=5x5, S=1x1, same) -> LeakyReLU -> Conv_Block(Depth=64, K=5x5, S=2x2) -> Conv_Block(Depth=128, K=5x5, S=2x2) -> Conv_Block(Depth=256, K=5x5, S=2x2) -> Flatten -> Dense(16384, Sigmoid)]
Also note that all Conv and Dense layers are initialized with the Truncated Normal distribution defined above. Another thing is that the bias term is removed from the Conv layers, which also stabilizes the model a little bit.
def DogDiscriminator(spectral_normalization=True):    model = Sequential()    if spectral_normalization:        model.add(ConvSN2D(64, (5, 5), strides=(1,1), padding='same', use_bias=False,                         input_shape=[image_height, image_width, image_channels],                          kernel_initializer=weight_initializer))        #model.add(BatchNormalization(epsilon=BN_EPSILON, momentum=BN_MOMENTUM))        model.add(LeakyReLU(alpha=leaky_relu_slope))        #model.add(Dropout(dropout_rate))                model = convSN(model, 64, ksize=5, stride_size=2)        #model = convSN(model, 128, ksize=3, stride_size=1)        model = convSN(model, 128, ksize=5, stride_size=2)        #model = convSN(model, 256, ksize=3, stride_size=1)        model = convSN(model, 256, ksize=5, stride_size=2)        #model = convSN(model, 512, ksize=3, stride_size=1)        #model.add(Dropout(dropout_rate))        model.add(Flatten())        model.add(DenseSN(1, activation='sigmoid'))    else:        ...    return modeldog_discriminator = DogDiscriminator(spectral_normalization=True)
One regularization method that can be applied during training is called Label smoothing. What this does is that it essentially prevents D from being overconfident or underconfident in its predictions. If D becomes too certain that there is a dog in a specific image, G can exploit that fact and continuously start to generate only images of that sort and in turn, cease to improve. We can combat this, by setting the class labels to be in the range [0, 0.3] for the negative classes and [0.7, 1] for the positive ones.
This will prevent the overall probabilities from getting very close to the two thresholds.
# Label smoothing -- technique from GAN hacks, instead of assigning 1/0 as class labels, we assign a random integer in range [0.7, 1.0] for positive class# and [0.0, 0.3] for negative classdef smooth_positive_labels(y):    return y - 0.3 + (np.random.random(y.shape) * 0.5)def smooth_negative_labels(y):    return y + np.random.random(y.shape) * 0.3
This technique is also called Instance Noise. By adding a small amount of error to the labels (let’s say 5%), this tends to make the true and predicted distributions more spread out and thus start to overlap with each other. This in turn makes fitting a custom distribution of generated images easier in the learning process.
Here is a good example of how the two distributions look like with these techniques:
Here is how we can implement Instance Noise:
# randomly flip some labelsdef noisy_labels(y, p_flip):    # determine the number of labels to flip    n_select = int(p_flip * int(y.shape[0]))    # choose labels to flip    flip_ix = np.random.choice([i for i in range(int(y.shape[0]))], size=n_select)        op_list = []    # invert the labels in place    #y_np[flip_ix] = 1 - y_np[flip_ix]    for i in range(int(y.shape[0])):        if i in flip_ix:            op_list.append(tf.subtract(1, y[i]))        else:            op_list.append(y[i])        outputs = tf.stack(op_list)    return outputs
The best proven optimization algorithm for this task is Adam with a standard learning rate of 0.0002 for both models and a beta of 0.5.
generator_optimizer = tf.train.AdamOptimizer(learning_rate=lr_initial_g, beta1=0.5)discriminator_optimizer = tf.train.AdamOptimizer(learning_rate=lr_initial_d, beta1=0.5)
Another new trend in optimizing GANs recently has been applying a Relativistic loss function as opposed to the standard one. Those functions measure the probability that the real data is more “realistic” than the generated data. One of the more popular relativistic function choices include RaLSGAN (Relativistic Average Least Squares), RaSGAN (Relativistic Average Standard) and RaHinge (Relativistic Hinge loss).
But before all of that, let’s define the standard GAN loss:
As we can observe, this is basically the standard Binary Crossentropy loss used for classification tasks or the Logistic Loss between the real and generated distributions. In Tensorflow, this can be defined as follows:
In comparison, here is what an RSGAN (Relativistic Standard) loss looks like:
In this case the task is different, to measure the similarity between the real (r) and fake (f) data distributions. RSGAN reaches the optimal point when D(x) = 0.5 (i.e. C(xr)=C(xf)). There are many relativistic loss function variants and they all contain different methods for measuring this similarity. In this project, I’ve tried out 3 of the variants that seemed to have the best documented MIFID score (RaLSGAN, RaSGAN and RaHinge). Feel free to try out different losses for yourself to see if you can improve the performance ;).
Here’s a big list of the most commonly used ones:
Throughout many trials for this particular problem, I didn’t find any increase in performance by switching to Relativistic losses, so I decided to stick with the standard GAN loss function, as it is much simpler to estimate, though in some cases these losses can really speed up convergence in your model.
Here is what the Discriminator loss function will look like with Label Smoothing and Instance Noise applied. It’s basically the sum of two sub-losses (Fake — in terms of G’s images, Real — in terms of the actual training images).
def discriminator_loss(real_output, fake_output, loss_func, apply_label_smoothing=True, label_noise=True):    if label_noise and apply_label_smoothing:        real_output_noise = noisy_labels(tf.ones_like(real_output), 0.05)        fake_output_noise = noisy_labels(tf.zeros_like(fake_output), 0.05)        real_output_smooth = smooth_positive_labels(real_output_noise)        fake_output_smooth = smooth_negative_labels(fake_output_noise)        if loss_func == 'gan':             real_loss = cross_entropy(tf.ones_like(real_output_smooth), real_output)            fake_loss = cross_entropy(tf.zeros_like(fake_output_smooth), fake_output)        else:... other loss function variants        loss = fake_loss + real_loss        return loss
And the Generator loss function (with Label Smoothing applied) is just a standard Logistic Loss:
def generator_loss(real_output, fake_output, loss_func, apply_label_smoothing=True):    if apply_label_smoothing:        fake_output_smooth = smooth_negative_labels(tf.ones_like(fake_output))        if loss_func == 'gan':            return cross_entropy(tf.ones_like(fake_output_smooth), fake_output)        else:... other loss function variants    return loss
Let’s also fix the number of epochs for training and the number of images to feed to the Generator for visualizing intermediate results.
EPOCHS = 280num_examples_to_generate = 64seed = tf.random.normal([num_examples_to_generate, noise_dim])
One training step of a DCGAN consists of three standard steps:
Forward prop — G creates a batch of fake images; this, alongside a batch of real images is fed to D.Calculate both G and D’s loss function.Backprop — compute gradients for G and D optimize the weights.
Forward prop — G creates a batch of fake images; this, alongside a batch of real images is fed to D.
Calculate both G and D’s loss function.
Backprop — compute gradients for G and D optimize the weights.
def train_step(images, loss_type='gan'):    noise = tf.random.normal([batch_size, noise_dim])    with tf.GradientTape() as gen_tape, tf.GradientTape() as disc_tape:        generated_images = dog_generator(noise, training=True)                real_output = dog_discriminator(images, training=True)        fake_output = dog_discriminator(generated_images, training=True)                gen_loss = generator_loss(real_output, fake_output, loss_type, apply_label_smoothing=True)        disc_loss = discriminator_loss(real_output, fake_output, loss_type,                                        apply_label_smoothing=True, label_noise=True)     gradients_of_generator = gen_tape.gradient(gen_loss, dog_generator.trainable_variables)    gradients_of_discriminator = disc_tape.gradient(disc_loss, dog_discriminator.trainable_variables)    generator_optimizer.apply_gradients(zip(gradients_of_generator, dog_generator.trainable_variables))    discriminator_optimizer.apply_gradients(zip(gradients_of_discriminator, dog_discriminator.trainable_variables))        return gen_loss, disc_loss
Let’s also define some functions to visualize the model losses by epoch and as a whole.
def plot_losses(G_losses, D_losses, all_gl, all_dl, epoch):    plt.figure(figsize=(10,5))    plt.title("Generator and Discriminator Loss - EPOCH {}".format(epoch))    plt.plot(G_losses,label="G")    plt.plot(D_losses,label="D")    plt.xlabel("Iterations")    plt.ylabel("Loss")    plt.legend()    ymax = plt.ylim()[1]    plt.show()        plt.figure(figsize=(10,5))    plt.plot(np.arange(len(all_gl)),all_gl,label='G')    plt.plot(np.arange(len(all_dl)),all_dl,label='D')    plt.legend()    #plt.ylim((0,np.min([1.1*np.max(all_gl),2*ymax])))    plt.title('All Time Loss')    plt.show()
We can also use the following function to plot a grid of the generated images.
def generate_and_save_images(model, epoch, test_input, rows, cols):    # Notice `training` is set to False.    # This is so all layers run in inference mode (batchnorm).    predictions = model(test_input, training=False)    fig = plt.figure(figsize=(14,14))    for i in range(predictions.shape[0]):        plt.subplot(rows, cols, i+1)        plt.imshow((predictions[i, :, :, :] * 127.5 + 127.5) / 255.)        plt.axis('off')             plt.subplots_adjust(wspace=0, hspace=0)    plt.savefig('image_at_epoch_{:04d}.png'.format(epoch))    plt.show()
To generate a single test image, we can also reuse the same method.
def generate_test_image(model, noise_dim=noise_dim):    test_input = tf.random.normal([1, noise_dim])    # Notice `training` is set to False.    # This is so all layers run in inference mode (batchnorm).    predictions = model(test_input, training=False)    fig = plt.figure(figsize=(5,5))    plt.imshow((predictions[0, :, :, :] * 127.5 + 127.5) / 255.)    plt.axis('off')     plt.show()
We still haven’t mentioned how a GAN is usually evaluated. Most research papers that use benchmarks to estimate how well a GAN performs are usually based on what’s called the Inception Score. This measures two main characteristics of the input images:
The variety (for example generating different types of dog breeds)
The distinction (or quality of the image)
If both things are true, the score will be high. If either or both are false, the score will be low. A higher score is better. It means your GAN can generate many different distinct images. The lowest score possible is zero. Mathematically the highest possible score is infinity, although in practice there will probably emerge a non-infinite ceiling.
The Inception Score is derived from Google’s Inception Network, which is one of the state-of-the-art deep architectures for classifying images. By passing the images from our GAN through the classifier, we can measure properties of our generated images. To produce the score, we need to calculate the similarity/distance between the real and fake distributions of images. This is done using the KL(Kullback–Leibler) divergence formula:
Here, P and Q are the two measured distributions. In this case, higher KL divergence means better results — the quality of the images is similar and there is a wide variety of labels present. In the opposite case, the low KL divergence can be due to either low quality or low variety of labels:
One shortcoming for IS is that it can misrepresent the performance if it only generates one image per class. To combat this we can use the FID (Frechet Inception Distance). This measure defines the two previous types of images as multivariate Gaussian distributions with mean μ and covariance Σ (Sigma). Let’s see how this distance is calculated:
Here, x and g represent the real and fake distribution of images, while Tr is the sum of diagonal elements of the results.
Lower FID values mean better image quality and diversity.
Here are also a few useful notes on why FID is a good measure:
FID is more robust to noise than IS.
If the model only generates one image per class, the distance will be high. So FID is a better measurement for image diversity.
By computing the FID between a training dataset and a testing dataset, we should expect the FID to be zero since both are real images. (although there is usually a small amount of error)
FID and IS are based on feature extraction (the presence or the absence of features).
Here is Kaggle’s official evaluation workflow for the Generative Dogs competition:
As we can see, in addition to the FID metric, there is an additional Memorization Score added to the calculation. This is basically a Cosine distance formula that measures the similarity between the real (images from a private dataset) and fake images. My guess is that this has been done to make sure that the images fed to the evaluation kernel have actually been generated by a GAN and not just replicated or modified from the real dataset.
Thankfully, the MIFID evaluator has been implemented by the Kaggle team (here) and we don’t have to worry about it.
I’m just going to add two more for zipping the final 10K images for submission and generating temporary images to calculate the MIFID in between certain epochs during training.
def zip_images(filename='images.zip'):    # SAVE TO ZIP FILE NAMED IMAGES.ZIP    z = zipfile.PyZipFile(filename, mode='w')    for k in range(image_sample_size):        generated_image = dog_generator(tf.random.normal([1, noise_dim]), training=False)        f = str(k)+'.png'        img = np.array(generated_image)        img = (img[0, :, :, :] + 1.) / 2.        img = Image.fromarray((255*img).astype('uint8').reshape((image_height,image_width,image_channels)))        img.save(f,'PNG')        z.write(f)        os.remove(f)        #if k % 1000==0: print(k)    z.close()    print('Saved final images for submission.')    def save_images(directory=OUT_DIR):    for k in range(image_sample_size):        generated_image = dog_generator(tf.random.normal([1, noise_dim]), training=False)        f = str(k)+'.png'        f = os.path.join(directory, f)        img = np.array(generated_image)        img = (img[0, :, :, :] + 1.) / 2.        img = Image.fromarray((255*img).astype('uint8').reshape((image_height,image_width,image_channels)))        img.save(f,'PNG')        #if k % 1000==0: print(k)    print('Saved temporary images for evaluation.')
It’s finally time to implement the final training function that wraps up the whole process. There are also a few techniques used here that I haven’t mentioned yet. Let’s see what they are.
This one is experimental and doesn’t always help improve performance, but I don’t think it would hurt either way. The idea here is to decrease the learning rate by a very small amount for each training step/steps, in order to stabilize the training process and speed up convergence (and escape from local minima). For this project, I am using the Cosine learning rate decay in Tensorflow to reduce the learning rate for every decay_step iterations.
Other than Non-convergence and Vanishing and Exploding gradients, GANs sometimes suffer from another major problem called Mode Collapse. This happens when G starts to produce limited varieties of samples. Here is a good example of Mode Collapse for a GAN trained on the MNIST dataset where G continuously produces only images for a single class label:
We’ve already seen some methods that can potentially eliminate Mode Collapse like Label smoothing, Instance Noise, Weight Initialization and so on. One other method that we could apply during training is called Experience Replay.
Experience Replay preserves some of the recently generated images in memory. For every replay_step iterations, we train D on those previous images to "remind" the network of previous generations and thus decrease the chance of overfitting to a particular instance of data batches during training. In this example, I am using a slightly different form of Experience Replay in the sense that, I am generating a new extra image for each training step to store in a list, instead of feeding it actual generated images from previous iterations, since storing data during Eager execution isn't an easy task.
'''        generated_image = dog_generator(tf.random.normal([1, noise_dim]), training=False)        exp_replay.append(generated_image)        if len(exp_replay) == replay_step:            print('Executing experience replay..')            replay_images = np.array([p[0] for p in exp_replay])            dog_discriminator(replay_images, training=True)            exp_replay = []    '''
EDIT: As Kaggle was running into memory issues after ~7–8 hours of runtime, I decided against using experience replay. Let me know if you find a workaround for this :D.
All in all, the training process is fairly straight-forward. There are additional steps for displaying the intermediate results like the images, losses and calculating the MIFID. At the end of the learning process, we print out the final evaluation and a larger grid of the final images.
display_results = 40calculate_mifid = 100replay_step = 50decay_step = 50def train(dataset, epochs):    all_gl = np.array([]); all_dl = np.array([])    for epoch in tqdm(range(epochs)):                G_loss = []; D_loss = []                start = time.time()        new_lr_d = lr_initial_d        new_lr_g = lr_initial_g        global_step = 0                for image_batch in dataset:            g_loss, d_loss = train_step(image_batch)            global_step = global_step + 1            G_loss.append(g_loss); D_loss.append(d_loss)            all_gl = np.append(all_gl,np.array([G_loss]))            all_dl = np.append(all_dl,np.array([D_loss]))        if (epoch + 1) % display_results == 0 or epoch == 0:            plot_losses(G_loss, D_loss, all_gl, all_dl, epoch + 1)            generate_and_save_images(dog_generator, epoch + 1, seed, rows=8, cols=8)                if (epoch + 1) % calculate_mifid == 0:                        OUT_DIR.mkdir(exist_ok=True)            save_images(OUT_DIR)            evaluator = MiFIDEvaluator(MODEL_PATH, TRAIN_DIR)            fid_value, distance, mi_fid_score = evaluator.evaluate(OUT_DIR)            print(f'FID: {fid_value:.5f}')            print(f'distance: {distance:.5f}')            print(f'MiFID: {mi_fid_score:.5f}')            shutil.rmtree(OUT_DIR)            print('Removed temporary image directory.')                # Cosine learning rate decay        if (epoch + 1) % decay_step == 0:            new_lr_d = tf.train.cosine_decay(new_lr_d, min(global_step, lr_decay_steps), lr_decay_steps)            new_lr_g = tf.train.cosine_decay(new_lr_g, min(global_step, lr_decay_steps), lr_decay_steps)            generator_optimizer = tf.train.AdamOptimizer(learning_rate=new_lr_d, beta1=0.5)            discriminator_optimizer = tf.train.AdamOptimizer(learning_rate=new_lr_g, beta1=0.5)                  print('Epoch: {} computed for {} sec'.format(epoch + 1, time.time() - start))        print('Gen_loss mean: ', np.mean(G_loss),' std: ', np.std(G_loss))        print('Disc_loss mean: ', np.mean(D_loss),' std: ', np.std(D_loss))    # Generate after the final epoch and repeat the process    generate_and_save_images(dog_generator, epochs, seed, rows=8, cols=8)    checkpoint.save(file_prefix = checkpoint_prefix)       OUT_DIR.mkdir(exist_ok=True)    save_images(OUT_DIR)    evaluator = MiFIDEvaluator(MODEL_PATH, TRAIN_DIR)    fid_value, distance, mi_fid_score = evaluator.evaluate(OUT_DIR)    print(f'FID: {fid_value:.5f}')    print(f'distance: {distance:.5f}')    print(f'MiFID: {mi_fid_score:.5f}')    shutil.rmtree(OUT_DIR)    print('Removed temporary image directory.')          print('Final epoch.')
Here are some of the generated dog images during training:
As we can observe, the MIFID steadily improves for 280 epochs (~ 8 hours). The best score that I achieved using this model during the competition was 55.87. The learning process does tend to be somewhat random so I think a score around the region of [50, 65] should be realistic. Feel free to train this model for longer, if you have the time, as it has potential to keep improving :).
Finally, I’ll show you how to make a fun GIF to see a nice little simulation of the DCGAN’s learning process (code from Tensorflow’s DCGAN tutorial).
anim_file = 'dcgan.gif'with imageio.get_writer(anim_file, mode='I') as writer:    filenames = glob.glob('image*.png')    filenames = sorted(filenames)    last = -1    for i,filename in enumerate(filenames):        frame = 1*(i**2)        if round(frame) > round(last):            last = frame        else:            continue        image = imageio.imread(filename)        writer.append_data(image)    image = imageio.imread(filename)    writer.append_data(image)import IPythonif IPython.version_info > (6,2,0,''):    IPython.display.Image(filename=anim_file)
To sum up, DCGANs seem to be extremely sensitive to hyperparameter choice and a lot of problems can arise during training, including Mode Collapse. They are also very computationally intensive and it is incredibly hard to build a high-scoring model for a runtime of ~9 hours. Fortunately, there is a whole bucket list of possible methods and techniques that are well-documented and can be easily applied to your model to stabilize the training process.
For me personally, it was really fun to toy around with these techniques and break a few kernels in the process :D. Feel free to leave any suggestions down in the comments (for improving the model or fixing something that I messed up).
Big thanks to Chris Deotte, Nanashi, Chad Malla and Nirjhar Roy for their Kaggle kernels and examples during the competition. I’ll leave links to them down below.
Overall, this was the first Kaggle competition that I’ve ever entered and it was an extremely fun method for learning about GANs and playing around with them. This seems to be Kaggle’s first competition involving generative modelling and let’s hope that more exciting challenges like this will come in the future ;).
[1]. My previous kernel on EDA and image preprocessing
[2]. Xml parsing and cropping to specified bounding box
[3]. Image cropping method with interpolation
[4]. Another great Keras-based DCGAN approach by Chad Malla
[5]. DCGAN hacks for improving your model performance
[6]. Tensorflow DCGAN tutorial
[7]. DCGAN Dogs Images by Nanashi
[8]. GAN dogs starter 24-Jul -Custom Layers by Nirjhar Roy
[9]. Supervised Generative Dog Net by Chris Deotte
[10]. My best submission for this competition
[1]. Ian Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, S. Ozair, Y. Bengio, Generative Adversarial Networks (2014)
[2]. J. Rocca, Understanding Generative Adversarial Networks (GANs)
[3]. J. Brownlee, A Gentle Introduction to Generative Adversarial Networks (GANs)
[4]. A. Radford, L. Metz, S. Chintala, Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks (2015)
[5]. J. Hui, GAN — DCGAN (Deep convolutional generative adversarial networks)
[6]. S. Yadav, Weight Initialization Techniques in Neural Networks
[7]. T. Miyato, T. Kataoka, M. Koyama, Y. Yoshida, Spectral Normalization for Generative Adversarial Networks (2018)
[8]. IShengFang, Spectral Normalization implemented in Keras
[9]. C. Cosgrove, Spectral Normalization Explained
[10]. J. Hui, GAN — Ways to improve GAN performance
[11]. J. Brownlee, How to Implement GAN Hacks in Keras to Train Stable Models
[12]. Y. Jiao, Tricks of GANS
[13]. C. K. Sønderby, Instance Noise: A trick for stabilising GAN training
[14]. J. Hui, GAN — RSGAN & RaGAN (A new generation of cost function.)
[15]. D. Mack, A simple explanation of the Inception Score
[16]. J. Hui, GAN — How to measure GAN performance?
[17]. All you need is GAN Hacks
[18]. How to train your touchy GANs — Things that seem to work.
[19]. Explaining the metric FID