Variational Auto Encoders. What happens when we encounter data... | by Viv | Towards Data Science

What happens when we encounter data with no labels? Most deep learning techniques require cleanly labelled data, but how realistic is this? At its core, this technique essentially says that if you have a set of inputs and their respective target labels, you can try and learn the probability of a particular label for a particular target. Sure, so glorified image mapping? In this post, I am exploring the variational autoencoder to probe a little deeper into the world of unlabelled data. This model will generate unique images after training on a collection of images with no labels.
Auto encoders sequentially de-construct input data into hidden representations and use these representations to sequentially reconstruct outputs that resemble their originals. It is essentially data compression that is data specific, meaning it can only compress that is similar to the data it has been trained on. Autoencoders are also known to be lossy, so the decompressed output will be a little degraded compared to the original inputs. Well so if they cause a loss in quality, why are they useful? The good question turns out, they are quite useful for data de-noising, where we train an autoencoder to reconstruct the input from a corrupted version of itself so that it can de-nise similar corrupted data.
In this post, I use my main man, Siraj’s support via his youtube channel and dive into the Variational Auto Encoder (VAE).
Let's start by talking about the Bayesian inference. Everyone reading this is probably aware of deep learning and its usefulness when it comes to approximating complex functions, however, the Bayesian inference offers a unique framework to reason uncertainty. In it, all uncertainty is expressed in terms of probability. This makes sense if you think about it, at any given time there exists evidence either in favour of or against something we already know, which can then be used to create a new probability. Extending this a little further, when we learn something new, we have to take into account what we already know and add this new evidence into consideration to create a new probability. The Bayesian theory basically describes this idea mathematically.
The VAE is a child of these ideas. Looking at it from a Bayesian standpoint, we can treat the inputs, hidden representations and reconstructed outputs of the VAE as probabilistic, random variables within a directed graphical model. Assuming it contains a specific probability model of some data, x, and a latent/hidden variable, z. We can write out the joint probability of the model like so:
Given just a character produced by the model, we wouldn’t know what setting of the latent variable has generated that character, our model is inherently stochastic!
Now, let's pull out the deep learning lens, a VAE consists of 3 main things:
An Encoder
A Decoder
A Loss Function
Given some input x, let's say we have a 28 by 28 handwritten digit image, which works out to 784 dimensions, where each pixel is one dimension. Now, this will encode into a latent/hidden representation space, which will be a lot less than 784. We can now sample a Gaussian probability density to get the noisy values of the representations.
Cool. Let's write this in code! (I have linked the code in its entirety at the end of this post)
First, we import the libraries and find our hyperparameters.
import numpy as npimport matplotlib.pyplot as pltfrom scipy.stats import normfrom keras.layers import Input, Dense, Lambdafrom keras.models import Modelfrom keras import backend as Kfrom keras import objectivesfrom keras.datasets import mnist# Hyperparametersbatch_size = 100original_dim = 784latent_dim = 2intermediate_dim = 256nb_epoch = 10epsilon_std = 1.0
Next, we initialize our encoder network. This network’s job is to map inputs to our hidden distribution parameters. We take the input and send it through a dense fully connected layer with ReLU (classic non-linearity to squash dimensionality). Next, we convert the input data into two parameters in the hidden space. We predefine the size using dense, fully connected layers — z mean and z log sigma.
#encoderx = Input(batch_shape = (batch_size, original_dim))h = Dense(intermediate_dim, activation = ‘relu’)(x)z_mean = Dense(latent_dim)(h)z_log_var = Dense(latent_dim)(h)print(z_mean)print(z_log_var)
The decoder will take ‘z’ as its input and output the parameters to the probability distribution of the data. So let's assume that each pixel is either a 1 or 0 (black or white), now we can use Bernoulli distribution as it will define success as a binary value to represent a single pixel. So the decoder will get the latent/hidden representation of a digit as its input, and it outputs 784 Bernoulli parameters, one for each pixel, so essentially 784 values between 0 and 1.
We will be using the z_mean and z_log_var to randomly sample new, similar points from the hidden/latent normal distribution by defining a sampling function. The epsilon in the code block below is a random normal tensor.
def sampling(args): z_mean, z_log_var = args epsilon = K.random_normal(shape=(batch_size, latent_dim), mean=0.) return z_mean + K.exp(z_log_var / 2) * epsilon # note that “output_shape” isn’t necessary with the TensorFlow backendz = Lambda(sampling, output_shape=(latent_dim,))([z_mean, z_log_var])#latent hidden stateprint(z)
Once, we get z, we can feed it to our decoder and the decoder will map these latent space points back to the original input data. So, to build a decoder we first initialize it with two fully connected layers and their own respective activation functions. Because the data is extracted from a small dimensionality to a larger one, some of it will be lost in the reconstruction process.
#decoder# we instantiate these layers separately so as to reuse them laterdecoder_h = Dense(intermediate_dim, activation=’relu’)decoder_mean = Dense(original_dim, activation=’sigmoid’)h_decoded = decoder_h(z)x_decoded_mean = decoder_mean(h_decoded)print(x_decoded_mean)
Cool, but how much are some? So we’ll be building our loss function to measure exactly this. The first term below measures the reconstruction loss. If the decoder output is bad at reconstructing data, the cost in terms of loss will be considered at this point. The next term is a regularizer, meaning it keeps the representation of each digit as diverse as possible. So for instance, if two different people were to write out the digit three, the representations might end up looking very different because, of course, different people write differently. This can be bad, and the regularizer comes to the rescue! We penalize bad behaviour (like the example here) and ensure similar representations are close together. Our total loss function is defined as the sum of our reconstruction term and the KL divergence regularization term.
#lossdef vae_loss(x, x_decoded_mean): xent_loss = original_dim * objectives.binary_crossentropy(x, x_decoded_mean) kl_loss = — 0.5 * K.sum(1 + z_log_var — K.square(z_mean) — K.exp(z_log_var), axis=-1) return xent_loss + kl_lossvae = Model(x, x_decoded_mean)vae.compile(optimizer=’rmsprop’, loss=vae_loss)
Now comes the training part, we would usually train this model by using gradient descent to optimize our loss with respect to the parameters of the encoder and decoder. But how are we going to take derivatives with respect to the parameters of a randomly determined variable?
Turns out, we’ve built randomness into our model itself. Now, gradient descent usually expects that a given input always returns the same output for a fixed set of parameters. The only source of randomness in our case would be the inputs. So how do we tackle this? We parametrize! We will parametrize the samples so that the randomness can be independent of the parameters.
We will define a function that depends on the parameters deterministically, and so we can inject randomness into the model by introducing a random variable. The encoder will generate a vector of means and a vector of standard deviations, instead of generating a vector of real values. We take derivatives of the functions involving z with respect to the parameters of its distribution. We have defined our model’s optimizer as rmsprop and the loss function as vae_loss.
We start the training below by importing the MNIST dataset and feeding them to our model for a given number of epochs and batch size.
# train the VAE on MNIST digits(x_train, y_train), (x_test, y_test) = mnist.load_data()x_train = x_train.astype(‘float32’) / 255.x_test = x_test.astype(‘float32’) / 255.x_train = x_train.reshape((len(x_train), np.prod(x_train.shape[1:])))x_test = x_test.reshape((len(x_test), np.prod(x_test.shape[1:])))vae.fit(x_train, x_train, shuffle=True, nb_epoch=nb_epoch, batch_size=batch_size, validation_data=(x_test, x_test),verbose=1)
Below we plot out the neighbourhoods on a 2D plane. Each coloured cluster represents a digit representation, and close clusters are essentially digits that are structurally similar.
# build a model to project inputs on the latent spaceencoder = Model(x, z_mean)# display a 2D plot of the digit classes in the latent spacex_test_encoded = encoder.predict(x_test, batch_size=batch_size)plt.figure(figsize=(6, 6))plt.scatter(x_test_encoded[:, 0], x_test_encoded[:, 1], c=y_test)plt.colorbar()plt.show()
Another way to represent this is to generate digits by scanning the latent plan, sampling latent points at regular intervals and generating the corresponding digit for each of these points like so:
# build a digit generator that can sample from the learned distributiondecoder_input = Input(shape=(latent_dim,))_h_decoded = decoder_h(decoder_input)_x_decoded_mean = decoder_mean(_h_decoded)generator = Model(decoder_input, _x_decoded_mean)# display a 2D manifold of the digitsn = 15 # figure with 15x15 digitsdigit_size = 28figure = np.zeros((digit_size * n, digit_size * n))# linearly spaced coordinates on the unit square were transformed through the inverse CDF (ppf) of the Gaussian# to produce values of the latent variables z, since the prior of the latent space is Gaussiangrid_x = norm.ppf(np.linspace(0.05, 0.95, n))grid_y = norm.ppf(np.linspace(0.05, 0.95, n))for i, yi in enumerate(grid_x): for j, xi in enumerate(grid_y): z_sample = np.array([[xi, yi]]) x_decoded = generator.predict(z_sample) digit = x_decoded[0].reshape(digit_size, digit_size) figure[i * digit_size: (i + 1) * digit_size, j * digit_size: (j + 1) * digit_size] = digitplt.figure(figsize=(10, 10))plt.imshow(figure, cmap=’Greys_r’)plt.show()
This has got to blow your mind at some level!
So at the essence of this exercise, there are three key takeaways:
Variational Encoders allow us to generate data by performing unsupervised learning
VAEs = Bayesian Inference + Deep Learning
Reparameterization allows us to backpropagate through the network, by making randomness independent of the parameters that allow us to derive our gradients.
This is a really fascinating world in the universe of machine learning and I hope I was able to bring some value to you through this post!