GAN — Introduction and Implementation — PART1: Implement a simple GAN in TF for MNIST handwritten digit generation | by Manish Chablani | Towards Data Science

The idea behind GANs is that you have two networks, a generator GG and a discriminator DD, competing against each other. The generator makes fake data to pass to the discriminator. The discriminator also sees real data and predicts if the data it’s received is real or fake. The generator is trained to fool the discriminator, it wants to output data that looks as close as possible to real data. And the discriminator is trained to figure out which data is real and which is fake. What ends up happening is that the generator learns to make data that is indistinguishable from real data to the discriminator.
This is equilibrium state and expectation is discriminator is emitting a probability of 0.5 for both real and fake data.
The general structure of a GAN is shown in the diagram above, using MNIST images as data. The latent sample is a random vector the generator uses to construct it’s fake images. As the generator learns through training, it figures out how to map these random vectors to recognizable images that can fool the discriminator.
The output of the discriminator is a sigmoid function, where 0 indicates a fake image and 1 indicates an real image. If you’re interested only in generating new images, you can throw out the discriminator after training.
https://github.com/mchablani/deep-learning/blob/master/gan_mnist/Intro_to_GANs_Exercises.ipynb
We use a leaky ReLU to allow gradients to flow backwards through the layer unimpeded. TensorFlow doesn’t provide an operation for leaky ReLUs, you can just take the outputs from a linear fully connected layer and pass them to tf.maximum. Typically, a parameter alpha sets the magnitude of the output for negative values. So, the output for negative input (x) values is alpha*x, and the output for positive x is x:
For the generator, we’re going to train it, but also sample from it as we’re training and after training. The discriminator will need to share variables between the fake and real input images. So, we can use the reuse keyword for tf.variable_scope to tell TensorFlow to reuse the variables instead of creating new ones if we build the graph again.
The generator has been found to perform the best with tanh for the generator output. This means that we’ll have to rescale the MNIST images to be between -1 and 1, instead of 0 and 1.
def generator(z, out_dim, n_units=128, reuse=False,  alpha=0.01):        with tf.variable_scope('generator', reuse=reuse):        # Hidden layer        h1 = tf.layers.dense(z, n_units, activation=None)        # Leaky ReLU        h1 = tf.maximum(h1, alpha*h1)                # Logits and tanh output        logits = tf.layers.dense(h1, out_dim, activation=None)        out = tf.nn.tanh(logits)                return out, logits
The discriminator network is almost exactly the same as the generator network, except that we’re using a sigmoid output layer.
def discriminator(x, n_units=128, reuse=False, alpha=0.01):    with tf.variable_scope('discriminator', reuse=reuse):        # Hidden layer        h1 = tf.layers.dense(x, n_units, activation=None)        # Leaky ReLU        h1 = tf.maximum(h1, alpha*h1)                logits = tf.layers.dense(h1, 1, activation=None)        out = tf.nn.sigmoid(logits)                return out, logits# Hyperparameters# Size of input image to discriminatorinput_size = 784 # 28x28 MNIST images flattened# Size of latent vector to generatorz_size = 100# Sizes of hidden layers in generator and discriminatorg_hidden_size = 128d_hidden_size = 128# Leak factor for leaky ReLUalpha = 0.01# Label smoothing smooth = 0.1
tf.reset_default_graph() # Create our input placeholders input_real, input_z = model_inputs(input_size, z_size)  # Generator network here g_model, g_logits = generator(input_z, input_size, g_hidden_size, reuse=False,  alpha=alpha) # g_model is the generator output  # Disriminator network here d_model_real, d_logits_real = discriminator(input_real, d_hidden_size, reuse=False, alpha=alpha) d_model_fake, d_logits_fake = discriminator(g_model, d_hidden_size, reuse=True, alpha=alpha)
For the discriminator, the total loss is the sum of the losses for real and fake images, d_loss = d_loss_real + d_loss_fake.
For the real image logits, we’ll use d_logits_real which we got from the discriminator in the cell above. For the labels, we want them to be all ones, since these are all real images. To help the discriminator generalize better, the labels are reduced a bit from 1.0 to 0.9, for example, using the parameter smooth. This is known as label smoothing, typically used with classifiers to improve performance. In TensorFlow, it looks something like labels = tf.ones_like(tensor) * (1 - smooth)
The discriminator loss for the fake data is similar. The logits are d_logits_fake, which we got from passing the generator output to the discriminator. These fake logits are used with labels of all zeros. Remember that we want the discriminator to output 1 for real images and 0 for fake images, so we need to set up the losses to reflect that.
Finally, the generator losses are using d_logits_fake, the fake image logits. But, now the labels are all ones. The generator is trying to fool the discriminator, so it wants to discriminator to output ones for fake images.
# Calculate lossesd_labels_real = tf.ones_like(d_logits_real) * (1 - smooth)d_labels_fake = tf.zeros_like(d_logits_fake)d_loss_real = tf.nn.sigmoid_cross_entropy_with_logits(labels=d_labels_real, logits=d_logits_real)d_loss_fake = tf.nn.sigmoid_cross_entropy_with_logits(labels=d_labels_fake, logits=d_logits_fake)d_loss = tf.reduce_mean(d_loss_real + d_loss_fake)g_loss = tf.reduce_mean(    tf.nn.sigmoid_cross_entropy_with_logits(        labels=tf.ones_like(d_logits_fake),         logits=d_logits_fake))
We want to update the generator and discriminator variables separately.
Note that when minimizing d_loss we want optimizer to only be updating the discriminator vars and similar for generator.
# Optimizerslearning_rate = 0.002# Get the trainable_variables, split into G and D partst_vars = tf.trainable_variables()g_vars = [var for var in t_vars if var.name.startswith("generator")]d_vars = [var for var in t_vars if var.name.startswith("discriminator")]d_train_opt = tf.train.AdamOptimizer().minimize(d_loss, var_list=d_vars)g_train_opt = tf.train.AdamOptimizer().minimize(g_loss, var_list=g_vars)
batch_size = 100epochs = 100samples = []losses = []saver = tf.train.Saver(var_list = g_vars)with tf.Session() as sess:    sess.run(tf.global_variables_initializer())    for e in range(epochs):        for ii in range(mnist.train.num_examples//batch_size):            batch = mnist.train.next_batch(batch_size)                        # Get images, reshape and rescale to pass to D            batch_images = batch[0].reshape((batch_size, 784))            batch_images = batch_images*2 - 1                        # Sample random noise for G            batch_z = np.random.uniform(-1, 1, size=(batch_size, z_size))                        # Run optimizers            _ = sess.run(d_train_opt, feed_dict={input_real: batch_images, input_z: batch_z})            _ = sess.run(g_train_opt, feed_dict={input_z: batch_z})                # At the end of each epoch, get the losses and print them out        train_loss_d = sess.run(d_loss, {input_z: batch_z, input_real: batch_images})        train_loss_g = g_loss.eval({input_z: batch_z})                    print("Epoch {}/{}...".format(e+1, epochs),              "Discriminator Loss: {:.4f}...".format(train_loss_d),              "Generator Loss: {:.4f}".format(train_loss_g))            # Save losses to view after training        losses.append((train_loss_d, train_loss_g))                # Sample from generator as we're training for viewing afterwards        sample_z = np.random.uniform(-1, 1, size=(16, z_size))        gen_samples = sess.run(                       generator(input_z, input_size, reuse=True),                       feed_dict={input_z: sample_z})        samples.append(gen_samples)        saver.save(sess, './checkpoints/generator.ckpt')# Save training generator sampleswith open('train_samples.pkl', 'wb') as f:    pkl.dump(samples, f)
Credits: From lecture notes: https://classroom.udacity.com/nanodegrees/nd101/syllabus