Title: GANs with Keras and TensorFlow - PyImageSearch

URL Source: https://pyimagesearch.com/2020/11/16/gans-with-keras-and-tensorflow/

Published Time: 2020-11-16T15:00:00+00:00

Markdown Content:
In this tutorial you will learn how to implement Generative Adversarial Networks (GANs) using Keras and TensorFlow.

![Image 1](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20700%20450'%3E%3C/svg%3E)

Generative Adversarial Networks were first introduced by Goodfellow et al. in their 2014 paper, _[Generative Adversarial Networks](https://arxiv.org/abs/1406.2661)._ These networks can be used to generate synthetic (i.e., fake) images that are perceptually near identical to their ground-truth authentic originals.

In order to generate synthetic images, we make use of _two_ neural networks during training:

1.  A **generator** that accepts an input vector of randomly generated noise and produces an output “imitation” image that looks similar, if not identical, to the authentic image
2.  A **discriminator** or **adversary** that attempts to determine if a given image is an “authentic” or “fake”

By training these networks at the same time, one giving feedback to the other, we can learn to generate synthetic images.

Inside this tutorial we’ll be implementing a variation of Radford et al.’s paper, _[Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks](https://arxiv.org/abs/1511.06434)_ — or more simply, **DCGANs.**

As we’ll find out, training GANs can be a _notoriously_ hard task, so we’ll implement a number of best practices recommended by both Radford et al. and Francois Chollet (creator of Keras and deep learning scientist at Google).

By the end of this tutorial, you’ll have a fully functioning GAN implementation.

**To learn how to implement Generative Adversarial Networks (GANs) with Keras and TensorFlow, _just keep reading._**

![Image 2](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20128%20128'%3E%3C/svg%3E)

#### Looking for the source code to this post?

[Jump Right To The Downloads Section](#download-the-code)

_**Note:** This tutorial is a chapter from my book [Deep Learning for Computer Vision with Python](https://pyimagesearch.com/deep-learning-computer-vision-python-book/). If you enjoyed this post and would like to learn more about deep learning applied to computer vision, be sure to give my book a read_ — _I have no doubt it will take you from deep learning beginner all the way to expert._

In the first part of this tutorial, we’ll discuss what Generative Adversarial Networks are, including how they are different from more “vanilla” network architectures you have seen before for classification and regression.

From there we’ll discuss the general GAN training process, including some guidelines and best practices you should follow when training your own GANs.

Next, we’ll review our directory structure for the project and then implement our GAN architecture using Keras and TensorFlow.

Once our GAN is implemented, we’ll train it on the Fashion MNIST dataset, thereby allowing us to generate fake/synthetic fashion apparel images.

Finally, we’ll wrap up this tutorial on Generative Adversarial Networks with a discussion of our results.

### **What are Generative Adversarial Networks (GANs)?**

![Image 3](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20700%20341'%3E%3C/svg%3E)

**Figure 1:** When training our GAN, the goal is for the generator to become progressively better and better at generating synthetic images, to the point where the discriminator is _unable to tell_ the difference between the real vs. synthetic data ([image source](https://developers.google.com/machine-learning/gan/gan_structure)).

The quintessential explanation of GANs typically involves some variant of two people working in collusion to forge a set of documents, replicate a piece of artwork, or print counterfeit money — the counterfeit money printers is my personal favorite, and the one used by Chollet in his work.

In this example, we have two people:

1.  Jack, the counterfeit printer (the _generator_)
2.  Jason, an employee of the U.S. Treasury (which is responsible for printing money in the United States), who specializes in detecting counterfeit money (the _discriminator_)

Jack and Jason were childhood friends, both growing up without much money in the rough parts of Boston. After much hard work, Jason was awarded a college scholarship — Jack was not, and over time started to turn toward illegal ventures to make money (in this case, creating counterfeit money).

Jack knew he wasn’t very good at generating counterfeit money, but he felt that with the proper training, he could replicate bills that were passable in circulation.

One day, after a few too many pints at a local pub during the Thanksgiving holiday, Jason let it slip to Jack that he wasn’t happy with his job. He was underpaid. His boss was nasty and spiteful, often yelling and embarrassing Jason in front of other employees. Jason was even thinking of quitting.

Jack saw an opportunity to use Jason’s access at the U.S. Treasury to create an elaborate counterfeit printing scheme. Their conspiracy worked like this:

1.  Jack, the counterfeit printer, would print fake bills and then mix _both_ the fake bills and real money together, then show them to the expert, Jason.
2.  Jason would sort through the bills, classifying each bill as “fake” or “authentic,” giving feedback to Jack along the way on how he could improve his counterfeit printing.

At first, Jack is doing a pretty poor job at printing counterfeit money. But over time, with Jason’s guidance, Jack eventually improves to the point where Jason is no longer able to spot the difference between the bills. By the end of this process, both Jack and Jason have stacks of counterfeit money that can fool most people.

### **The general GAN training procedure**

![Image 4](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20693%20468'%3E%3C/svg%3E)

**Figure 2:** The steps involved in training a Generative Adversarial Network (GAN) with Keras and TensorFlow.

We’ve discussed what GANs are in terms of an analogy, but what is the actual _procedure_ to train them? Most GANs are trained using a six-step process.

To start (Step 1), we randomly generate a vector (i.e., noise). We pass this noise through our generator, which generates an actual image (Step 2). We then sample authentic images from our training set and mix them with our synthetic images (Step 3).

The next step (Step 4) is to train our discriminator using this mixed set. The goal of the discriminator is to correctly label each image as “real” or “fake.”

Next, we’ll once again generate random noise, but this time we’ll purposely label each noise vector as a “real image” (Step 5). We’ll then train the GAN using the noise vectors and “real image” labels even though they are not actual real images (Step 6).

The reason this process works is due to the following:

1.  We have frozen the weights of the discriminator at this stage, implying that the discriminator is not learning when we update the weights of the generator.
2.  We’re trying to “fool” the discriminator into being unable to determine which images are real vs. synthetic. The feedback from the discriminator will allow the generator to learn how to produce more authentic images.

If you’re confused with this process, I would continue reading through our implementation covered later in this tutorial — seeing a GAN implemented in Python and then explained makes it easier to understand the process.

### **Guidelines and best practices when training GANs**

![Image 5](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20700%20394'%3E%3C/svg%3E)

**Figure 3:** Generative Adversarial Networks are incredibly hard to train due to the evolving loss landscape. Here are some tips to help you successfully train your GANs ([image source](https://www.xenonstack.com/images/insights/2019/12/generative-adversarial-networks-applications-xenonstack.png)).

GANs are _notoriously hard to train_ due to an **evolving loss landscape**. At each iteration of our algorithm we are:

1.  Generating random images and then training the discriminator to correctly distinguish the two
2.  Generating additional synthetic images, but this time purposely trying to fool the discriminator
3.  Updating the weights of the generator based on the feedback of the discriminator, thereby allowing us to generate more authentic images

From this process you’ll notice there are _two losses_ we need to observe: one loss for the discriminator and a second loss for the generator. And since the loss landscape of the generator can be changed based on the feedback from the discriminator, _we end up with a dynamic system._

**When training GANs, our goal is not to seek a minimum loss value but instead to find some equilibrium between the two (Chollet 2017).**

This concept of finding an equilibrium may make sense on paper, but once you try to implement and train your own GANs, you’ll find that this is a nontrivial process.

**In their [paper](https://arxiv.org/abs/1511.06434), Radford et al. recommend the following architecture guidelines for more stable GANs:**

*   Replace any pooling layers with strided convolutions (see [this tutorial](https://pyimagesearch.com/2018/12/31/keras-conv2d-and-convolutional-layers/) for more information on convolutions and strided convolutions).
*   Use batch normalization in both the generator and discriminator.
*   Remove fully-connected layers in deeper networks.
*   Use ReLU in the generator except for the final layer, which will utilize _tanh._
*   Use Leaky ReLU in the discriminator.

**In his book, Francois Chollet then provides additional recommendations on training GANs:**

1.  Sample random vectors from a _normal distribution_ (i.e., Gaussian distribution) rather than a _uniform distribution._
2.  Add dropout to the discriminator.
3.  Add noise to the class labels when training the discriminator.
4.  To reduce checkerboard pixel artifacts in the output image, use a kernel size that is divisible by the stride when utilizing convolution or transposed convolution in both the generator and discriminator.
5.  If your adversarial loss rises dramatically while your discriminator loss falls to zero, try reducing the learning rate of the discriminator and increasing the dropout of the discriminator.

Keep in mind that these are all just _heuristics_ found to work in a number of situations — we’ll be using _some_ of the techniques suggested by both Radford et al. and Chollet, but not _all_ of them.

It is possible, and even _probable_, that the techniques listed here will not work on your GANs. Take the time now to set your expectations that you’ll likely be running _orders of magnitude more experiments_ when tuning the hyperparameters of your GANs as compared to more basic classification or regression tasks.

### **Configuring your development environment to train GANs with Keras and TensorFlow**

We’ll be using Keras and TensorFlow to implement and train our GANs.

I recommend you follow either of these two guides to install TensorFlow and Keras on your system:

*   _[How to install TensorFlow 2.0 on Ubuntu](https://pyimagesearch.com/2019/12/09/how-to-install-tensorflow-2-0-on-ubuntu/)_
*   _[How to install TensorFlow 2.0 on macOS](https://pyimagesearch.com/2019/12/09/how-to-install-tensorflow-2-0-on-macos/)_

Either tutorial will help you configure your system with all the necessary software for this blog post in a convenient Python virtual environment.

### **Having problems configuring your development environment?**

[![Image 6](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20500%20334'%3E%3C/svg%3E)](https://pyimagesearch.com/pyimagesearch-plus/)

**Figure 4:** Having trouble configuring your dev environment? Want access to pre-configured Jupyter Notebooks running on Google Colab? Be sure to join [PyImageSearch Plus](https://pyimagesearch.com/pyimagesearch-plus/) —- you’ll be up and running with this tutorial in a matter of minutes.

All that said, are you:

*   Short on time?
*   Learning on your employer’s administratively locked system?
*   Wanting to skip the hassle of fighting with the command line, package managers, and virtual environments?
*   **Ready to run the code _right now_ on your Windows, macOS, or Linux system?**

Then join [PyImageSearch Plus](https://pyimagesearch.com/pyimagesearch-plus/) today!

**Gain access to Jupyter Notebooks for this tutorial and other PyImageSearch guides that are _pre-configured_ to run on Google Colab’s ecosystem right in your web browser!** No installation required.

And best of all, these Jupyter Notebooks will run on Windows, macOS, and Linux!

### **Project structure**

Now that we understand the fundamentals of Generative Adversarial Networks, let’s review our directory structure for the project.

Make sure you use the **_“Downloads”_** section of this tutorial to download the source code to our GAN project:

$ tree . --dirsfirst
.
├── output
│   ├── epoch\_0001\_output.png
│   ├── epoch\_0001\_step\_00000.png
│   ├── epoch\_0001\_step\_00025.png
...
│   ├── epoch\_0050\_step\_00300.png
│   ├── epoch\_0050\_step\_00400.png
│   └── epoch\_0050\_step\_00500.png
├── pyimagesearch
│   ├── \_\_init\_\_.py
│   └── dcgan.py
└── dcgan\_fashion\_mnist.py

3 directories, 516 files

The `dcgan.py` file inside the `pyimagesearch` module contains the implementation of our GAN in Keras and TensorFlow.

The `dcgan_fashion_mnist.py` script will take our GAN implementation and train it on the [Fashion MNIST dataset](https://pyimagesearch.com/2019/02/11/fashion-mnist-with-keras-and-deep-learning/), thereby allowing us to generate “fake” examples of clothing using our GAN.

The output of the GAN after every set number of steps/epochs will be saved to the `output` directory, allowing us to visually monitor and validate that the GAN is learning how to generate fashion items.

### **Implementing our “generator” with Keras and TensorFlow**

Now that we’ve reviewed our project directory structure, let’s get started implementing our Generative Adversarial Network using Keras and TensorFlow.

Open up the `dcgan.py` file in our project directory structure, and let’s get started:

\# import the necessary packages
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import BatchNormalization
from tensorflow.keras.layers import Conv2DTranspose
from tensorflow.keras.layers import Conv2D
from tensorflow.keras.layers import LeakyReLU
from tensorflow.keras.layers import Activation
from tensorflow.keras.layers import Flatten
from tensorflow.keras.layers import Dense
from tensorflow.keras.layers import Reshape

**Lines 2-10** import our required Python packages. All of these classes should look fairly familiar to you, especially if you’ve read my [_Keras and TensorFlow_ tutorials](https://pyimagesearch.com/category/keras-and-tensorflow/) or my book [_Deep Learning for Computer Vision with Python._](https://pyimagesearch.com/deep-learning-computer-vision-python-book/)

The only exception may be the `Conv2DTranspose` class. Transposed convolutional layers, sometimes referred to as _fractionally-strided convolution_ or (incorrectly) _deconvolution_, are used when we need a transform going in the _opposite direction_ of a normal convolution.

The generator of our GAN will accept an _N_ dimensional input vector (i.e., a _list_ of numbers, but a _volume_ like an image) and then transform the _N_ dimensional vector into an output image.

This process implies that we need to _reshape_ and then _upscale_ this vector into a volume as it passes through the network — to accomplish this reshaping and upscaling, we’ll need transposed convolution.

We can thus look at transposed convolution as the method to:

1.  Accept an input volume from a previous layer in the network
2.  Produce an output volume that is _larger_ than the input volume
3.  Maintain a connectivity pattern between the input and output

In essence our transposed convolution layer will reconstruct our target spatial resolution and perform a normal convolution operation, utilizing fancy zero-padding techniques to ensure our output spatial dimensions are met.

To learn more about transposed convolution, take a look at the [_Convolution arithmetic tutorial_ in the Theano documentation](http://deeplearning.net/software/theano/tutorial/conv_arithmetic.html) along with _[An introduction to different Types of Convolutions in Deep Learning](https://towardsdatascience.com/types-of-convolutions-in-deep-learning-717013397f4d)_ By Paul-Louis Pröve.

Let’s now move into implementing our `DCGAN` class:

class DCGAN:
	@staticmethod
	def build\_generator(dim, depth, channels=1, inputDim=100,
		outputDim=512):
		# initialize the model along with the input shape to be
		# "channels last" and the channels dimension itself
		model = Sequential()
		inputShape = (dim, dim, depth)
		chanDim = -1

Here we define the `build_generator` function inside `DCGAN`. The `build_generator` accepts a number of arguments:

*   `` `dim` ``: The target spatial dimensions (width and height) of the generator after reshaping
*   `` `depth` ``: The target depth of the volume after reshaping
*   `` `channels` ``: The number of channels in the output volume from the generator (i.e., `1` for grayscale images and `3` for RGB images)
*   `` `inputDim` ``: Dimensionality of the randomly generated input vector to the generator
*   `` `outputDim` ``: Dimensionality of the output fully-connected layer from the randomly generated input vector

The usage of these parameters will become more clear as we define the body of the network in the next code block.

**Line 19** defines the `inputShape` of the volume after we reshape it from the fully-connected layer.

**Line 20** sets the channel dimension (`chanDim`), which we assume to be “channels-last” ordering (the standard channel ordering for TensorFlow).

Below we can find the body of our generator network:

		# first set of FC => RELU => BN layers
		model.add(Dense(input\_dim=inputDim, units=outputDim))
		model.add(Activation("relu"))
		model.add(BatchNormalization())

		# second set of FC => RELU => BN layers, this time preparing
		# the number of FC nodes to be reshaped into a volume
		model.add(Dense(dim \* dim \* depth))
		model.add(Activation("relu"))
		model.add(BatchNormalization())

**Lines 23-25** define our first set of `FC => RELU => BN` layers — applying batch normalization to stabilize GAN training is a guideline from Radford et al. (see the _“Guidelines and best practices when training GANs”_ section above).

Notice how our `FC` layer will have an input dimension of `inputDim` (the randomly generated input vector) and then output dimensionality of `outputDim`. Typically `outputDim` will be _larger_ than `inputDim`.

**Lines 29-31** apply a second set of `FC => RELU => BN` layers, but this time we prepare the number of nodes in the `FC` layer to equal the number of units in `inputShape` (**Line 29**). Even though we are still utilizing a flattened representation, we need to ensure the output of this `FC` layer can be reshaped to our target volume sze (i.e., `inputShape`).

The actual reshaping takes place in the next code block:

		# reshape the output of the previous layer set, upsample +
		# apply a transposed convolution, RELU, and BN
		model.add(Reshape(inputShape))
		model.add(Conv2DTranspose(32, (5, 5), strides=(2, 2),
			padding="same"))
		model.add(Activation("relu"))
		model.add(BatchNormalization(axis=chanDim))

A call to `Reshape` while supplying the `inputShape` allows us to create a 3D volume from the fully-connected layer on **Line 29**. **Again, this reshaping is only possible due to the fact that the number of output nodes in the `FC` layer matches the target `inputShape`.**

We now reach an important guideline when training your own GANs:

1.  To _increase_ spatial resolution, use a transposed convolution with a stride _\> 1._
2.  To create a deeper GAN _without_ increasing spatial resolution, you can use either standard convolution or transposed convolution (but keep the stride equal to 1).

Here, our transposed convolution layer is learning `32` filters, each of which is _5×5_, while applying a _2×2_ stride — since our stride is _\> 1_, we can increase our spatial resolution.

Let’s apply another transposed convolution:

		# apply another upsample and transposed convolution, but
		# this time output the TANH activation
		model.add(Conv2DTranspose(channels, (5, 5), strides=(2, 2),
			padding="same"))
		model.add(Activation("tanh"))

		# return the generator model
		return model

**Lines 43 and 44** apply another transposed convolution, again increasing the spatial resolution, but taking care to ensure the number of filters learned is equal to the target number of `channels` (`1` for grayscale and `3` for RGB).

We then apply a _tanh_ activation function per the recommendation of Radford et al. The model is then returned to the calling function on **Line 48.**

### **Understanding the “generator” in our GAN**

Assuming `dim=7`, `depth=64`, `channels=1`, `inputDim=100`, and `outputDim=512` (as we will use when training our GAN on Fashion MNIST later in this tutorial), I have included the model summary below:

Model: "sequential"
\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_
Layer (type)                 Output Shape              Param #   
=================================================================
dense (Dense)                (None, 512)               51712     
\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_
activation (Activation)      (None, 512)               0         
\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_
batch\_normalization (BatchNo (None, 512)               2048      
\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_
dense\_1 (Dense)              (None, 3136)              1608768   
\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_
activation\_1 (Activation)    (None, 3136)              0         
\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_
batch\_normalization\_1 (Batch (None, 3136)              12544     
\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_
reshape (Reshape)            (None, 7, 7, 64)          0         
\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_
conv2d\_transpose (Conv2DTran (None, 14, 14, 32)        51232     
\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_
activation\_2 (Activation)    (None, 14, 14, 32)        0         
\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_
batch\_normalization\_2 (Batch (None, 14, 14, 32)        128       
\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_
conv2d\_transpose\_1 (Conv2DTr (None, 28, 28, 1)         801       
\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_
activation\_3 (Activation)    (None, 28, 28, 1)         0        
================================================================= 

Let’s break down what’s going on here.

First, our model will accept an input vector that is 100-d, then transform it to a 512-d vector via an `FC` layer.

We then add a second `FC` layer, this one with _7x7x64 = 3,136_ nodes. We reshape these 3,136 nodes into a 3D volume with shape _7×7 = 64_ — this reshaping is only possible since our previous `FC` layer matches the number of nodes in the reshaped volume.

Applying a transposed convolution with a _2×2_ stride increases our spatial dimensions from _7×7_ to _14×14._

A second transposed convolution (again, with a stride of _2×2_) increases our spatial dimension resolution from _14×14_ to _28×18_ with a single channel, which is the _exact_ dimensions of our input images in the Fashion MNIST dataset.

**When implementing your own GANs, make sure the spatial dimensions of the output volume match the spatial dimensions of your input images.** Use transposed convolution to increase the spatial dimensions of the volumes in the generator. I also recommend using `model.summary()` often to help you debug the spatial dimensions.

### **Implementing our “discriminator” with Keras and TensorFlow**

The discriminator model is _substantially_ more simplistic, similar to basic CNN classification architectures you may have read [in my book](https://pyimagesearch.com/deep-learning-computer-vision-python-book/) or elsewhere on the PyImageSearch blog.

Keep in mind that while the generator is intended to _create_ synthetic images, the discriminator is used to _classify_ whether any given input image is _real_ or _fake._

Continuing our implementation of the `DCGAN` class in `dcgan.py`, let’s take a look at the discriminator now:

	@staticmethod
	def build\_discriminator(width, height, depth, alpha=0.2):
		# initialize the model along with the input shape to be
		# "channels last"
		model = Sequential()
		inputShape = (height, width, depth)

		# first set of CONV => RELU layers
		model.add(Conv2D(32, (5, 5), padding="same", strides=(2, 2),
			input\_shape=inputShape))
		model.add(LeakyReLU(alpha=alpha))

		# second set of CONV => RELU layers
		model.add(Conv2D(64, (5, 5), padding="same", strides=(2, 2)))
		model.add(LeakyReLU(alpha=alpha))

		# first (and only) set of FC => RELU layers
		model.add(Flatten())
		model.add(Dense(512))
		model.add(LeakyReLU(alpha=alpha))

		# sigmoid layer outputting a single value
		model.add(Dense(1))
		model.add(Activation("sigmoid"))

		# return the discriminator model
		return model

As we can see, this network is simple and straightforward. We first learn 32, _5×5_ filters, followed by a second `CONV` layer, this one learning a total of 64, _5×5_ filters. We only have a single `FC` layer here, this one with `512` nodes.

All activation layers utilize a Leaky ReLU activation to stabilize training, _except_ for the final activation function which is sigmoid. We use a sigmoid here to capture the probability of whether the input image is _real_ or _synthetic._

### **Implementing our GAN training script**

Now that we’ve implemented our DCGAN architecture, let’s train it on the Fashion MNIST dataset to generate fake apparel items. By the end of the training process, we will be unable to identify _real_ images from _synthetic_ ones.

Open up the `dcgan_fashion_mnist.py` file in our project directory structure, and let’s get to work:

\# import the necessary packages
from pyimagesearch.dcgan import DCGAN
from tensorflow.keras.models import Model
from tensorflow.keras.layers import Input
from tensorflow.keras.optimizers import Adam
from tensorflow.keras.datasets import fashion\_mnist
from sklearn.utils import shuffle
from imutils import build\_montages
import numpy as np
import argparse
import cv2
import os

We start off by importing our required Python packages.

Notice that we’re importing `DCGAN`, which is our implementation of the GAN architecture from the previous section (**Line 2**).

We also import the `build_montages` function (**Line 8**). This is a convenience function that will enable us to easily build a montage of generated images and then display them to our screen as a single image. You can read more about building montages in my tutorial _[Montages with OpenCV](https://pyimagesearch.com/2017/05/29/montages-with-opencv/)._

Let’s move to parsing our command line arguments:

\# construct the argument parse and parse the arguments
ap = argparse.ArgumentParser()
ap.add\_argument("-o", "--output", required=True,
	help="path to output directory")
ap.add\_argument("-e", "--epochs", type=int, default=50,
	help="# epochs to train for")
ap.add\_argument("-b", "--batch-size", type=int, default=128,
	help="batch size for training")
args = vars(ap.parse\_args())

We require only a single command line argument for this script, `--output`, which is the path to the output directory where we’ll store montages of generated images (thereby allowing us to visualize the GAN training process).

We can also (optionally) supply `--epochs`, the total number of epochs to train for, and `--batch-size`, used to control the batch size when training.

Let’s now take care of a few important initializations:

\# store the epochs and batch size in convenience variables, then
# initialize our learning rate
NUM\_EPOCHS = args\["epochs"\]
BATCH\_SIZE = args\["batch\_size"\]
INIT\_LR = 2e-4

We store both the number of epochs and batch size in convenience variables on **Lines 26 and 27**.

We also initialize our initial learning rate (`INIT_LR`) on **Line 28.** This value was _empirically tuned_ through a number of experiments and trial and error. If you choose to apply this GAN implementation to your own dataset, you may need to tune this learning rate.

We can now load the Fashion MNIST dataset from disk:

\# load the Fashion MNIST dataset and stack the training and testing
# data points so we have additional training data
print("\[INFO\] loading MNIST dataset...")
((trainX, \_), (testX, \_)) = fashion\_mnist.load\_data()
trainImages = np.concatenate(\[trainX, testX\])

# add in an extra dimension for the channel and scale the images
# into the range \[-1, 1\] (which is the range of the tanh
# function)
trainImages = np.expand\_dims(trainImages, axis=-1)
trainImages = (trainImages.astype("float") - 127.5) / 127.5

**Line 33** loads the Fashion MNIST dataset from disk. We ignore class labels here, since we do not need them — we are only interested in the actual pixel data.

Furthermore, there is no concept of a “test set” for GANs. Our goal when training a GAN isn’t minimal loss or high accuracy. **Instead, we seek an equilibrium between the generator and the discriminator.**

To help us obtain this equilibrium, we _combine_ both the training and testing images (**Line 34**) to give us additional training data.

**Lines 39 and 40** prepare our data for training by scaling the pixel intensities to the range _\[0, 1\]_, the output range of the _tanh_ activation function.

Let’s now initialize our generator and discriminator:

\# build the generator
print("\[INFO\] building generator...")
gen = DCGAN.build\_generator(7, 64, channels=1)

# build the discriminator
print("\[INFO\] building discriminator...")
disc = DCGAN.build\_discriminator(28, 28, 1)
discOpt = Adam(lr=INIT\_LR, beta\_1=0.5, decay=INIT\_LR / NUM\_EPOCHS)
disc.compile(loss="binary\_crossentropy", optimizer=discOpt)

**Line 44** initializes the generator that will transform the input random vector to a volume of shape _7x7x64-_channel map.

**Lines 48-50** build the discriminator and then compile it using the Adam optimizer with binary cross-entropy loss.

Keep in mind that we are using _binary_ cross-entropy here, as our discriminator has a sigmoid activation function that will return a probability indicating whether the input image is real vs. fake. Since there are only two “class labels” (real vs. synthetic), we use binary cross-entropy.

The learning rate and beta value for the Adam optimizer were experimentally tuned. I’ve found that a lower learning rate and beta value for the Adam optimizer improves GAN training on the Fashion MNIST dataset. Applying learning rate decay helps stabilize training as well.

Given both the generator and discriminator, we can build our GAN:

\# build the adversarial model by first setting the discriminator to
# \*not\* be trainable, then combine the generator and discriminator
# together
print("\[INFO\] building GAN...")
disc.trainable = False
ganInput = Input(shape=(100,))
ganOutput = disc(gen(ganInput))
gan = Model(ganInput, ganOutput)

# compile the GAN
ganOpt = Adam(lr=INIT\_LR, beta\_1=0.5, decay=INIT\_LR / NUM\_EPOCHS)
gan.compile(loss="binary\_crossentropy", optimizer=discOpt)

The actual GAN consists of _both_ the generator and the discriminator; however, we first need to _freeze_ the discriminator weights (**Line 56**) before we combine the models to form our Generative Adversarial Network (**Lines 57-59**).

Here we can see that the input to the `gan` will take a random vector that is 100-d. This value will be passed through the generator first, the output of which will go to the discriminator — we call this “model composition,” similar to “function composition” we learned about back in algebra class.

**The discriminator weights are frozen at this point so the feedback from the discriminator will enable the generator to _learn_ how to generate better synthetic images.**

**Lines 62 and 63** compile the `gan`. I again use the Adam optimizer with the same hyperparameters as the optimizer for the discriminator — this process worked for the purposes of these experiments, but you may need to tune these values on your own datasets and models.

Additionally, I’ve often found that setting the learning rate of the GAN to be _half_ that of the discriminator is often a good starting point.

Throughout the training process we’ll want to see how our GAN evolves to construct synthetic images from random noise. To accomplish this task, we’ll need to generate some benchmark random noise used to visualize the training process:

\# randomly generate some benchmark noise so we can consistently
# visualize how the generative modeling is learning
print("\[INFO\] starting training...")
benchmarkNoise = np.random.uniform(-1, 1, size=(256, 100))

# loop over the epochs
for epoch in range(0, NUM\_EPOCHS):
	# show epoch information and compute the number of batches per
	# epoch
	print("\[INFO\] starting epoch {} of {}...".format(epoch + 1,
		NUM\_EPOCHS))
	batchesPerEpoch = int(trainImages.shape\[0\] / BATCH\_SIZE)

	# loop over the batches
	for i in range(0, batchesPerEpoch):
		# initialize an (empty) output path
		p = None

		# select the next batch of images, then randomly generate
		# noise for the generator to predict on
		imageBatch = trainImages\[i \* BATCH\_SIZE:(i + 1) \* BATCH\_SIZE\]
		noise = np.random.uniform(-1, 1, size=(BATCH\_SIZE, 100))

**Line 68** generates our `benchmarkNoise`. Notice that the `benchmarkNoise` is generated from a uniform distribution in the range _\[-1, 1\]_, the same range as our _tanh_ activation function. **Line 68** indicates that we’ll be generating 256 synthetic images, where each input starts as a 100-d vector.

Starting on **Line 71** we loop over our desired number of epochs. **Line 76** computes the number of batches per epoch by dividing the number of training images by the supplied batch size.

We then loop over each batch on **Line 79**.

**Line 85** subsequently extracts the next `imageBatch`, while **Line 86** generates the random noise that we’ll be passing through the generator.

Given the `noise` vector, we can use the generator to generate synthetic images:

		# generate images using the noise + generator model
		genImages = gen.predict(noise, verbose=0)

		# concatenate the \*actual\* images and the \*generated\* images,
		# construct class labels for the discriminator, and shuffle
		# the data
		X = np.concatenate((imageBatch, genImages))
		y = (\[1\] \* BATCH\_SIZE) + (\[0\] \* BATCH\_SIZE)
		y = np.reshape(y, (-1,))
		(X, y) = shuffle(X, y)

		# train the discriminator on the data
		discLoss = disc.train\_on\_batch(X, y)

**Line 89** takes our input `noise` and then generates synthetic apparel images (`genImages`).

Given our generated images, we need to train the discriminator to recognize the difference between real and synthetic images.

To accomplish this task, **Line 94** concatenates the current `imageBatch` and the synthetic `genImages` together.

We then need to build our class labels on **Line 95** — each _real_ image will have a class label of `1`, while every fake image will be labeled `0`.

The concatenated training data is then jointly shuffled on **Line 97** so our real and fake images do not sequentially follow each other one-by-one (which would cause problems during our gradient update phase).

Additionally, I have found this shuffling process improves the stability of discriminator training.

**Line 100** trains the discriminator on the current (shuffled) batch.

The final step in our training process is to train the `gan` itself:

		# let's now train our generator via the adversarial model by
		# (1) generating random noise and (2) training the generator
		# with the discriminator weights frozen
		noise = np.random.uniform(-1, 1, (BATCH\_SIZE, 100))
		fakeLabels = \[1\] \* BATCH\_SIZE
		fakeLabels = np.reshape(fakeLabels, (-1,))
		ganLoss = gan.train\_on\_batch(noise, fakeLabels)

We first generate a total of `BATCH_SIZE` random vectors. However, unlike in our previous code block, where we were nice enough to tell our discriminator what is real vs. fake, we’re now going to attempt to trick the discriminator by labeling the random `noise` as real images.

The feedback from the discriminator enables us to actually train the generator (keeping in mind that the discriminator weights are frozen for this operation).

Not only is looking at the loss values important when training a GAN, but you _also_ need to examine the output of the `gan` on your `benchmarkNoise`:

		# check to see if this is the end of an epoch, and if so,
		# initialize the output path
		if i == batchesPerEpoch - 1:
			p = \[args\["output"\], "epoch\_{}\_output.png".format(
				str(epoch + 1).zfill(4))\]

		# otherwise, check to see if we should visualize the current
		# batch for the epoch
		else:
			# create more visualizations early in the training
			# process
			if epoch < 10 and i % 25 == 0:
				p = \[args\["output"\], "epoch\_{}\_step\_{}.png".format(
					str(epoch + 1).zfill(4), str(i).zfill(5))\]

			# visualizations later in the training process are less
			# interesting
			elif epoch >= 10 and i % 100 == 0:
				p = \[args\["output"\], "epoch\_{}\_step\_{}.png".format(
					str(epoch + 1).zfill(4), str(i).zfill(5))\]

If we have reached the end of the epoch, we’ll build the path, `p`, to our output visualization (**Lines 112-114**).

Otherwise, I find it helpful to visually inspect the output of our GAN with more frequency in _earlier_ steps rather than _later_ ones (**Lines 118-129**).

The output visualization will be totally random salt and pepper noise at the beginning but should quickly start to develop characteristics of the input data. These characteristics may not look real, but the evolving attributes will demonstrate to you that the network is actually learning.

If your output visualizations are still salt and pepper noise after 5-10 epochs, it may be a sign that you need to tune your hyperparameters, potentially including the model architecture definition itself.

Our final code block handles writing the synthetic image visualization to disk:

		# check to see if we should visualize the output of the
		# generator model on our benchmark data
		if p is not None:
			# show loss information
			print("\[INFO\] Step {}\_{}: discriminator\_loss={:.6f}, "
				"adversarial\_loss={:.6f}".format(epoch + 1, i,
					discLoss, ganLoss))

			# make predictions on the benchmark noise, scale it back
			# to the range \[0, 255\], and generate the montage
			images = gen.predict(benchmarkNoise)
			images = ((images \* 127.5) + 127.5).astype("uint8")
			images = np.repeat(images, 3, axis=-1)
			vis = build\_montages(images, (28, 28), (16, 16))\[0\]

			# write the visualization to disk
			p = os.path.sep.join(p)
			cv2.imwrite(p, vis)

**Line 141** uses our generator to generate `images` from our `benchmarkNoise`. We then scale our image data back from the range _\[-1, 1\]_ (the boundaries of the _tanh_ activation function) to the range _\[0, 255\]_ (**Line 142**).

Since we are generating single-channel images, we repeat the grayscale representation of the image three times to construct a 3-channel RGB image (**Line 143**).

The `build_montages` function generates a _16×16_ grid, with a _28×28_ image in each vector. The montage is then written to disk on **Line 148**.

### **Training our GAN with Keras and TensorFlow**

To train our GAN on the Fashion MNIST dataset, make sure you use the **_“Downloads”_** section of this tutorial to download the source code.

From there, open up a terminal, and execute the following command:

$ python dcgan\_fashion\_mnist.py --output output
\[INFO\] loading MNIST dataset...
\[INFO\] building generator...
\[INFO\] building discriminator...
\[INFO\] building GAN...
\[INFO\] starting training...
\[INFO\] starting epoch 1 of 50...
\[INFO\] Step 1\_0: discriminator\_loss=0.683195, adversarial\_loss=0.577937
\[INFO\] Step 1\_25: discriminator\_loss=0.091885, adversarial\_loss=0.007404
\[INFO\] Step 1\_50: discriminator\_loss=0.000986, adversarial\_loss=0.000562
...
\[INFO\] starting epoch 50 of 50...
\[INFO\] Step 50\_0: discriminator\_loss=0.472731, adversarial\_loss=1.194858
\[INFO\] Step 50\_100: discriminator\_loss=0.526521, adversarial\_loss=1.816754
\[INFO\] Step 50\_200: discriminator\_loss=0.500521, adversarial\_loss=1.561429
\[INFO\] Step 50\_300: discriminator\_loss=0.495300, adversarial\_loss=0.963850
\[INFO\] Step 50\_400: discriminator\_loss=0.512699, adversarial\_loss=0.858868
\[INFO\] Step 50\_500: discriminator\_loss=0.493293, adversarial\_loss=0.963694
\[INFO\] Step 50\_545: discriminator\_loss=0.455144, adversarial\_loss=1.128864

![Image 7](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20700%20700'%3E%3C/svg%3E)

**Figure 5: Top-left:** The initial random noise of 256 input noise vectors. **Top-right:** The same random noise after two epochs. We are starting to see the makings of clothes/apparel items. **Bottom-left:** We are now starting to do a good job generating synthetic images based on training on the Fashion MNIST dataset. **Bottom-right:** The final fashion/apparel items after 50 epochs look _very_ authentic and realistic.

**Figure 5** shows our random noise vectors (i.e., `benchmarkNoise` during different moments of training):

*   The _top-left_ contains 256 (in an _8×8_ grid) of our initial random noise vectors before even starting to train the GAN. **We can clearly see there is no pattern in this noise.** No fashion items have been learned by the GAN.
*   However, by the end of the second epoch _(top-right),_ apparel-like structures are starting to appear.
*   By the end of the fifth epoch (_bottom-left_), the fashion items are _significantly_ more clear.
*   And by the time we reach the end of the 50th epoch (_bottom-right),_ our fashion items look authentic.

**Again, it’s important to understand that these fashion items are generated from random noise input vectors — _they are totally synthetic images!_**

### What's next? We recommend [PyImageSearch University](https://pyimagesearch.com/pyimagesearch-university/?utm_source=blogPost&utm_medium=bottomBanner&utm_campaign=What%27s%20next%3F%20I%20recommend).

**Course information:**  
84 total classes • 114+ hours of on-demand code walkthrough videos • Last updated: February 2024  
★★★★★ 4.84 (128 Ratings) • 16,000+ Students Enrolled

**I strongly believe that if you had the right teacher you could _master_ computer vision and deep learning.**

Do you think learning computer vision and deep learning has to be time-consuming, overwhelming, and complicated? Or has to involve complex mathematics and equations? Or requires a degree in computer science?

That’s _not_ the case.

All you need to master computer vision and deep learning is for someone to explain things to you in _simple, intuitive_ terms. _And that’s exactly what I do_. My mission is to change education and how complex Artificial Intelligence topics are taught.

If you're serious about learning computer vision, your next stop should be PyImageSearch University, the most comprehensive computer vision, deep learning, and OpenCV course online today. Here you’ll learn how to _successfully_ and _confidently_ apply computer vision to your work, research, and projects. Join me in computer vision mastery.

**Inside PyImageSearch University you'll find:**

*   ✓ **84 courses** on essential computer vision, deep learning, and OpenCV topics
*   ✓ **84 Certificates** of Completion
*   ✓ **114+ hours** of on-demand video
*   ✓ **Brand new courses released _regularly_**, ensuring you can keep up with state-of-the-art techniques
*   ✓ **Pre-configured Jupyter Notebooks in Google Colab**
*   ✓ Run all code examples in your web browser — works on Windows, macOS, and Linux (no dev environment configuration required!)
*   ✓ Access to **centralized code repos for _all_ 536+ tutorials** on PyImageSearch
*   ✓ **Easy one-click downloads** for code, datasets, pre-trained models, etc.
*   ✓ **Access** on mobile, laptop, desktop, etc.

[Click here to join PyImageSearch University](https://pyimagesearch.com/pyimagesearch-university/?utm_source=blogPost&utm_medium=bottomBanner&utm_campaign=What%27s%20next%3F%20I%20recommend)

**Summary**
-----------

In this tutorial we discussed Generative Adversarial Networks (GANs). We learned that GANs actually consist of _two_ networks:

1.  A **generator** that is responsible for generating fake images
2.  A **discriminator** that tries to spot the synthetic images from the authentic ones

By training both of these networks at the same time, we can learn to generate very realistic output images.

We then implemented [Deep Convolutional Adversarial Networks (DCGANS)](https://arxiv.org/abs/1511.06434), a variation of [Goodfellow et al.’s original GAN implementation.](https://arxiv.org/abs/1406.2661)

Using our DCGAN implementation, we trained both the generator and discriminator on the Fashion MNIST dataset, resulting in output images of fashion items that:

1.  Are _not_ part of the training set and are _complete synthetic_
2.  Look nearly identical to and indistinguishable from any image in the Fashion MNIST dataset

The problem is that training GANs can be _extremely_ challenging, more so than _any_ other architecture or method we have discussed on the PyImageSearch blog.

The reason GANs are notoriously hard to train is due to the _evolving loss landscape_ — with every step, our loss landscape changes slightly and is thus ever-evolving.

The evolving loss landscape is in stark contrast to other classification or regression tasks where the loss landscape is “fixed” and nonmoving.

When training your own GANs, you’ll undoubtedly have to carefully tune your model architecture and associated hyperparameters — be sure to refer to the _“Guidelines and best practices when training GANs”_ section at the top of this tutorial to help you tune your hyperparameters and run your own GAN experiments.

**To download the source code to this post (and be notified when future tutorials are published here on PyImageSearch), _simply enter your email address in the form below!_**

![Image 8](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20410%20567'%3E%3C/svg%3E)

#### Download the Source Code and FREE 17-page Resource Guide

Enter your email address below to get a .zip of the code and a **FREE 17-page Resource Guide on Computer Vision, OpenCV, and Deep Learning.** Inside you'll find my hand-picked tutorials, books, courses, and libraries to help you master CV and DL!