Semi-supervised learning with Generative Adversarial Networks (GANs) | by Thalles Silva | Towards Data Science

If you ever heard or studied about deep learning, you probably heard about MNIST, SVHN, ImageNet, PascalVoc and others. Each of these datasets has one thing in common. They consist of hundreds and thousands of labeled data. In other words, these collections are composed of (x,y) pairs where (x) is the raw data, an image matrix for instance, and (y) is a description of what that data point (x) represents.
Take the MNIST dataset as an example. Each of the 60,000 data points is a (input, label) pair. input is a 28x28 grayscale image and label is a sign representing what the input is. For the MNIST dataset, an input image may be a zero, one, two, three and so on up to 10 possible categories.
The most common usage of datasets like these is to develop supervised models. To train such algorithms, we usually provide a tremendous amount of data samples.
Supervised learning has been the center of most researching in deep learning. But, the necessity of creating models capable of learning from fewer data is increasing faster.
With that in mind, semi-supervised learning is a technique in which both labeled and unlabeled data are used to train a classifier.
This type of classifier takes a tiny portion of labeled data and a much larger amount of unlabeled data (from the same domain). The goal is to combine these sources of data to train a Deep Convolution Neural Networks (DCNN) to learn an inferred function capable of mapping a new datapoint to its desirable outcome.
In this frontier, we present a GAN model to classify street view house numbers using a very small labeled training set. In fact, the model uses roughly 1.3% of the original SVHN training labels i.e. 1000 (one thousand) labeled examples. We use some of the techniques described in the paper Improved Techniques for Training GANs from OpenAI.
If you are not familiar with image generation GANs, refer to A Short Introduction to Generative Adversarial Networks. This article refers to some of the contents described in that piece. The complete code notebook can found here.
When building a GAN for generating images, we trained both — the generator and the discriminator at the same time. After training, we could discard the discriminator because we used it only for training the generator.
For semi-supervised learning, we need to transform the discriminator into a multi-class classifier. This new model has to be able to generalize well on the test set, even through we do not have many labeled examples for training.
Additionally, this time, by the end of training, we can actually throw away the generator. Note that the roles changed. Now, the generator is only used for helping the discriminator during training.
Putting it differently, the generator acts as a different source of information from which the discriminator gets raw unlabeled training data. As we will see, this unlabelled data is key to improve the discriminator performance.
Also, for a regular image generation GAN, the discriminator has only one role. Compute the probability of whether its inputs are real or not — let’s call it the GAN problem.
However, to turn the discriminator into a semi-supervised classifier, besides to the GAN problem, the discriminator also has to learn the probabilities of each of the original dataset classes.
In other words, for each input image, the discriminator has to learn the probabilities of it being a one, two, three and so on.
Recall that for an image generation GAN discriminator, we have a single sigmoid unit output. This value represents the probability of an input image being real (value close to 1), or fake (value near 0 ).
Put in other words, from the discriminator point of view, values close to 1 means that the samples are likely to come from the training set. Likewise, value near 0 means a higher change that the samples come from the generator network.
By using this probability, the discriminator is able to send a signal back to the generator. This signal allows the generator to adapt its parameters during training making it possible to improve its capabilities of creating realistic images.
We have to convert the discriminator (from the previous GAN) into an 11 class classifier. To do that, we can turn its sigmoid output into a softmax with 11 class outputs. The first 10 for the individual class probabilities of the SVHN dataset (zero to nine), and the 11th class for all the fake images that come from the generator.
Note that if we set the 11th class probability to 0, then the sum of the first 10 probabilities represents the same probability computed using the sigmoid function.
Finally, we need to setup the losses in such a way that the discriminator can do both:
- (i) help the generator learning to produce realistic images. To do that, we have to instruct the discriminator to distinguish between real and fake samples.
- (ii) use the generator’s images, along with the labeled and unlabeled training data, to help classify the dataset.
To summarize, the discriminator has three different sources of training data.
Real images with labels. These are image label pairs like in any regular supervised classification problem.
Real images without labels. For those, the classifier only learns that these images are real.
Images from the generator. To these ones, the discriminator learns to classify as fake.
The combination of these different sources of data will make the classifier able to learn from a broader perspective. That in turn, allows the model to perform inference much more precisely than it would be if only using the 1000 labeled examples for training.
The Generator follows a very standard implementation described in the DCGAN paper. This approach consists of taking a random vector z as input. Reshape it to a 4D tensor and then feed it to a sequence of transpose convolutions, Batch Normalization (BN) and leaky ReLU operations.
This sequence of computations increases the spatial dimensions of the input vector while reduces its number of channels. As a result, the network outputs a 32x32x3 RGB tensor shape squashed between values of -1 and 1 through the Hyperbolic Tangent Function.
def generator(z, output_dim, reuse=False, alpha=0.2, training=True, size_mult=128):    with tf.variable_scope('generator', reuse=reuse):        # First fully connected layer        x1 = tf.layers.dense(z, 4 * 4 * size_mult * 4)        # Reshape it to start the convolutional stack        x1 = tf.reshape(x1, (-1, 4, 4, size_mult * 4))        x1 = tf.layers.batch_normalization(x1, training=training)        x1 = tf.maximum(alpha * x1, x1)        x2 = tf.layers.conv2d_transpose(x1, size_mult * 2, 5, strides=2, padding='same')        x2 = tf.layers.batch_normalization(x2, training=training)        x2 = tf.maximum(alpha * x2, x2)        x3 = tf.layers.conv2d_transpose(x2, size_mult, 5, strides=2, padding='same')        x3 = tf.layers.batch_normalization(x3, training=training)        x3 = tf.maximum(alpha * x3, x3)        # Output layer        logits = tf.layers.conv2d_transpose(x3, output_dim, 5, strides=2, padding='same')        out = tf.tanh(logits)        return out
The discriminator, now a multi-class classifier, is the most relevant network. Here, we setup an also similar DCGAN architecture in which we use a stack of convolutions with BN and ReLU.
We use strided convolutions for reducing the dimensions of the feature-vectors. Note that not all convolutions perform this type of computation. When we want to keep the feature vector’s dimensions intact, we use strides of 1 otherwise, we use strides of 2. Finally, to stabilize learning we make extensive use of BN (except the first layer of the network).
The 2D convolution window (kernel or filter) is set to have a width and height of 3 across all the convolutions. Also, note that we have some layers with dropout. It is important to understand that our discriminator behaves (in part) like any other regular classifier. Because of that, it may suffer from the same problems any classifier would if not well designed.
When training a big classifier on a very limited dataset, one of the most likely drawbacks one might encounter, is the immense of overfitting. One thing to watch on “over trained” classifiers is that they typically show a notably difference between the training error (smaller) and the testing error (higher).
This situation shows that the model did a good job capturing the structure of the training dataset. However, because it believed too much in the training data, it fails to generalize for unseen examples.
To prevent that, we make an extensive usage of regularization through dropout. Even for the first layer of the network.
def discriminator(x, reuse=False, alpha=0.2, drop_rate=0., num_classes=10, size_mult=64):    with tf.variable_scope('discriminator', reuse=reuse):        x = tf.layers.dropout(x, rate=drop_rate/2.5)        # Input layer is ?x32x32x3        x1 = tf.layers.conv2d(x, size_mult, 3, strides=2, padding='same')        relu1 = tf.maximum(alpha * x1, x1)        relu1 = tf.layers.dropout(relu1, rate=drop_rate) # [?x16x16x?]        x2 = tf.layers.conv2d(relu1, size_mult, 3, strides=2, padding='same')        bn2 = tf.layers.batch_normalization(x2, training=True) # [?x8x8x?]        relu2 = tf.maximum(alpha * bn2, bn2)        x3 = tf.layers.conv2d(relu2, size_mult, 3, strides=2, padding='same') # [?x4x4x?]        bn3 = tf.layers.batch_normalization(x3, training=True)        relu3 = tf.maximum(alpha * bn3, bn3)        relu3 = tf.layers.dropout(relu3, rate=drop_rate)        x4 = tf.layers.conv2d(relu3, 2 * size_mult, 3, strides=1, padding='same') # [?x4x4x?]        bn4 = tf.layers.batch_normalization(x4, training=True)        relu4 = tf.maximum(alpha * bn4, bn4)        x5 = tf.layers.conv2d(relu4, 2 * size_mult, 3, strides=1, padding='same') # [?x4x4x?]        bn5 = tf.layers.batch_normalization(x5, training=True)        relu5 = tf.maximum(alpha * bn5, bn5)        x6 = tf.layers.conv2d(relu5, 2 * size_mult, 3, strides=2, padding='same') # [?x2x2x?]        bn6 = tf.layers.batch_normalization(x6, training=True)        relu6 = tf.maximum(alpha * bn6, bn6)        relu6 = tf.layers.dropout(relu6, rate=drop_rate)...
In the end, instead of applying a fully connected layer on top of the convolution stack, we perform Global Average Pooling (GAP). In GAP, we take the average over the spatial dimensions of a feature vector. This operation results in squashing the tensor dimensions to a single value.
...# Flatten it by global average pooling# In global average pooling, for every feature map we take the average over all the spatial# domain and return a single value# In: [BATCH_SIZE,HEIGHT X WIDTH X CHANNELS] --> [BATCH_SIZE, CHANNELS]features = tf.reduce_mean(relu7, axis=[1,2])# Set class_logits to be the inputs to a softmax distribution over the different classesclass_logits = tf.layers.dense(features, num_classes)...
For instance, suppose that after a series of convolutions, we get a tensor of shape [BATCH_SIZE, 8, 8, NUM_CHANNELS]. To apply GAP, we take the average value over the [8x8] tensor slice. This results in a tensor of shape [BATCH_SIZE, 1, 1, NUM_CHANNELS] that can be reshaped to [BATCH_SIZE, NUM_CHANNELS].
In Network in Network, the authors describe some advantages of GAP over traditional fully connected layers. These include: higher robustness for spatial translation and less overfitting concerns. After GAP, we apply a fully connected layer to output the final logits. These have shape [BATCH_SIZE, NUM_CLASSES] and corresponds to the unscaled final class values.
To get the classification probabilities, we feed the logits through the softmax function. However, we still need a way to represent the probability of an input image being real rather than fake. That is, we still need to account for the binary classification problem of regular a GAN.
...# Get the probability that the input is real rather than fakeout = tf.nn.softmax(class_logits) # class probabilities for the 10 real classes...
We know that the logits are in terms of softmax probability values. Yet, we need a way to represent them as sigmoid logits as well. We know that the probability of an input being real corresponds to the sum over all real class logits. With that mind, we can feed these values to a LogSumExp function that will model the binary classification value. After that, we feed the result from the LogSumExp to a sigmoid function.
We can use the Tensorflow’s LogSumExp built-in function to avoid numerical problems. This routine prevents over/under flow issues that may occur when LogSumExp encounters very extreme, either positive or negative values.
...# This function is more numerically stable than log(sum(exp(input))).# It avoids overflows caused by taking the exp of large inputs and underflows# caused by taking the log of small inputs.gan_logits = tf.reduce_logsumexp(class_logits, 1)...
As we mentioned, we can divide the discriminator loss in two parts. One that represents the GAN problem, the unsupervised loss. And the other that computes the individual real class probabilities, the supervised loss.
For the unsupervised loss, the discriminator has to differentiate between real training images and fake images from the generator.
As for a regular GAN, half of the time the discriminator receives unlabeled images from the training set and the other half, imaginary unlabeled images from the generator.
In both cases we are dealing with a binary classification problem. Since we want a probability value near 1 for real images and close to 0 for unreal ones, we can use the sigmoid cross entropy function to compute the loss.
For images coming from the training set, we maximize their probabilities of being real by assigning labels of 1s. For fabricated images coming from the generator, we maximize their probabilities to be fake by giving them labels of 0s.
...# Here we compute `d_loss`, the loss for the discriminator.# This should combine two different losses:# 1. The loss for the GAN problem, where we minimize the cross-entropy for the binary#    real-vs-fake classification problem.tf.reduce_mean(tf.nn.sigmoid_cross_entropy_with_logits(logits=gan_logits_on_data,                                                        labels=tf.ones_like(gan_logits_on_data) * (1 - smooth)))fake_data_loss = tf.reduce_mean(tf.nn.sigmoid_cross_entropy_with_logits(logits=gan_logits_on_samples,                                                     labels=tf.zeros_like(gan_logits_on_samples)))# This way, the unsupervisedunsupervised_loss = real_data_loss + fake_data_loss...
For the supervised loss, we need to use the logits from the discriminator. Since this is a multi-class classification problem, we can use the softmax cross entropy function with the real labels we have available.
Note that this part is similar to any other classification model. In the end, the discriminator loss is the sum of both the supervised loss and the unsupervised loss. Also, because we are pretending we do not have most of the labels, we need to ignore them in the supervised loss. To do that, we multiply the loss by the masks variable that indicates which set of labels are available for usage.
#  2. The loss for the SVHN digit classification problem, where we minimize the cross-entropy#     for the multi-class softmax. For this one we use the labels. Don't forget to ignore#     use `label_mask` to ignore the examples that we are pretending are unlabeled for the#     semi-supervised learning problem.y = tf.squeeze(y)suppervised_loss = tf.nn.softmax_cross_entropy_with_logits(logits=class_logits_on_data,                                                              labels=tf.one_hot(y, num_classes, dtype=tf.float32))label_mask = tf.squeeze(tf.to_float(label_mask))# ignore the labels that we pretend does not exist for the losssuppervised_loss = tf.reduce_sum(tf.multiply(suppervised_loss, label_mask))# get the meansuppervised_loss = suppervised_loss / tf.maximum(1.0, tf.reduce_sum(label_mask))d_loss = unsupervised_loss + suppervised_loss
As described in the Improved Techniques for Training GANs paper, we use feature matching for the generator loss.
As the authors describe:
Feature matching is the concept of penalizing the mean absolute error between the average value of some set of features on the training data and the average values of that set of features on the generated samples.
To do that, we take some set of statistics (the moments) from two different sources and force them to be similar.
First, we take the average of the features extracted from the discriminator when a real training minibatch is being processed.
Second, we compute the moments in the same way, but now for when a minibatch composed of fake images that come from the generator was being analyzed by the discriminator.
Finally, with these two sets of moments, the generator loss is the mean absolute difference between them. In other words, as the paper emphasizes:
We train the generator to match the expected values of the features on an intermediate layer of the discriminator.
# Here we set `g_loss` to the "feature matching" loss invented by Tim Salimans at OpenAI.# This loss consists of minimizing the absolute difference between the expected features# on the data and the expected features on the generated samples.# This loss works better for semi-supervised learning than the tradition GAN losses.# Make the Generator output features that are on average similar to the features# that are found by applying the real data to the discriminatordata_moments = tf.reduce_mean(data_features, axis=0)sample_moments = tf.reduce_mean(sample_features, axis=0)g_loss = tf.reduce_mean(tf.abs(data_moments - sample_moments))pred_class = tf.cast(tf.argmax(class_logits_on_data, 1), tf.int32)eq = tf.equal(tf.squeeze(y), pred_class)correct = tf.reduce_sum(tf.to_float(eq))masked_correct = tf.reduce_sum(label_mask * tf.to_float(eq))
Although feature-matching loss performs well on the task of semi-supervised learning, the images produced by generator are not as good as the ones created in the last post.
In the Improved Techniques for Training GANs paper, OpenAI reports state-of-the-art results for semi-supervised classification learning on MNIST, CIFAR-10 and SVHN.
Our implementation reaches train and test accuracies of nearly 93% and 68% respectively. These results are better than the ones reported in this NIPS 2014 paper which got something around 64%.
This notebook is not intended for demonstrating best practices such cross-validation techniques. It only uses some of the techniques described in the original OpenAI paper.
The notebook was based on the Udacity Deep learning Fundamentals nanodegree program in which I graduated from.
Many researches considers unsupervised learning as the missing link to general AI systems.
To break these obstacles, attempts to solve already established problems using less labeled data are key. In this scenario, GANs pose a real alternative for learning complicated tasks with less labeled samples.
Yet, the performance gap between supervised and semi-supervised learning is still far from being equal. But we certainly can expect this gap to become shorter as new approaches come in to play.
If you are curious to dig deeper in these subjects, take a look at:
medium.freecodecamp.org
medium.freecodecamp.org
And if you need more, that is my deep learning blog.
Thanks for reading!
Credits to Sam Williams for this awesome “clap” gif! Check it out in his post.