What is a Variational Autoencoder? | by Roman Paolucci | Towards Data Science

Generative machine learning can be helpful in a variety of contexts. One, for example, is when some closed-form or quasi-closed-form solution is available but computationally expensive. In this case, a neural network may provide a timely solution with an acceptable degree of accuracy. Nevertheless, to train such a network, data must be aggregated and in most cases can be scarce. A great example can be found in the setting of pricing exotic options. I’ll leave a link to the SSRN paper if you wish to learn more about this specific example...
papers.ssrn.com
A solution to this problem can be found in generative machine learning via variational autoencoders. With such a model, we can learn from a representative set of population data and generate new samples which can serve as more data for training. This article will first consider autoencoders and give an intuitive example and transition to variational autoencoders with code laid out herein using Python and Tensorflow’s functional API.
The best way to understand a variational autoencoder is to first understand a standard autoencoder. If you are unfamiliar with standard neural network architecture I would advise you see these articles to get you up to speed...
Artificial Intelligence Bootcamp
Linear Regression v.s. Neural Networks
Assuming you have a foundation in the neural network space we can proceed with the structure of an autoencoder.
Generally, a feed-forward neural network will have an input layer, a certain number of hidden layers in the same, higher-dimensional, or lower-dimensional space than the inputs, and some target(s) in an output layer of lower-dimensional space relative to the hidden layers and inputs.
This notion is relatively straightforward, and anyone with experience constructing neural networks may immediately propose counterexamples to this generality which is more than fair, nevertheless, I digress. The purpose of this example is to highlight the dimensionality of each component of the neural network as an autoencoder has a specific structure that makes it an autoencoder.
The autoencoder takes an input (whether it be a grid of volatility points, pixels of an image, or the signature of a time-series path) and compresses it to a lower-dimensional representation called the latent space. From this lower-dimensional representation, the latent space, the goal is to reconstruct the inputs through backpropagation of reconstruction error.
This can be seen in a similar light to any sort of dimensionality reduction methodology (i.e. principal component analysis and path signatures).
As for the semantics, the autoencoder can be decomposed into two models:
The Encoder: compresses inputs to a latent space and is comprised of the Inputs and Hidden Layers in the diagram above
The Decoder: takes the latent representation of the original inputs as an input and attempts to reconstruct the original inputs
There aren’t many resources on implementing autoencoders in Tensorflow so I’ve included a gist below to serve as a guide for developing your own autoencoders using Tensorflow’s functional API.
Model: "encoder"_________________________________________________________________Layer (type)                 Output Shape              Param #   =================================================================encoder_input (InputLayer)   (None, 10)                0         _________________________________________________________________intermediate_layer_1 (Dense) (None, 10)                110       _________________________________________________________________intermediate_layer_2 (Dense) (None, 10)                110       _________________________________________________________________latent_space (Dense)         (None, 5)                 55        =================================================================Total params: 275Trainable params: 275Non-trainable params: 0_________________________________________________________________
Now we need a decoder...
Model: "decoder"_________________________________________________________________Layer (type)                 Output Shape              Param #   =================================================================decoder_input (InputLayer)   (None, 5)                 0         _________________________________________________________________intermediate_layer_3 (Dense) (None, 10)                60        _________________________________________________________________latent_space (Dense)         (None, 10)                110       =================================================================Total params: 170Trainable params: 170Non-trainable params: 0_________________________________________________________________
Now that we have both the encoder and decoder we can compile them into one model called (shockingly) an autoencoder.
Model: "autoencoder"_________________________________________________________________Layer (type)                 Output Shape              Param #   =================================================================encoder_input (InputLayer)   (None, 10)                0         _________________________________________________________________encoder (Model)              (None, 5)                 275       _________________________________________________________________decoder (Model)              (None, 10)                170       =================================================================Total params: 445Trainable params: 445Non-trainable params: 0_________________________________________________________________
That’s all there is to it! We’ve successfully built a dynamic autoencoder using Python and Tensorflow. Now that we have a solid foundation in autoencoders, let’s see why variational autoencoders are so useful.
As the name implies the only difference between this and a standard autoencoder is the variational component...
Autoencoders have a discrete representation of the original inputs in the latent space. Discrete, meaning they aren’t probabilistic in nature, they are mathematically certain outcomes.
Variational autoencoders maintain a distribution in the latent space. If it sounds confusing that’s because it is. Let’s see a diagram of this architecture to shed some light on this notion...
Where X in the diagram is referring to an instance of an input in a forward pass. We can see the latent space is comprised of a distribution of latent variables which is directly related to the mean and standard deviation developed by the encoder (recall the structure from our autoencoder, this still holds).
In the case of a variational autoencoder, the encoder develops a conditional mean and standard deviation that is responsible for constructing the distribution of latent variables. Without these conditional means and standard deviations, the decoder would have no frame of reference for reconstructing the original input.
To create the distribution itself, we add an epsilon term (distributed normally) which is multiplied by the conditional standard deviation and summed with the conditional mean. This is to bypass the backpropagation issue through a random node. Though beyond the scope of this article, if you are interested as to why we must use this reparameterization trick you may see the following post.
This is beginning to get very technical so keep the original goal in mind: generate new samples representative of some original population of data.
Persisting this notion of a distribution of latent variables means we can generate new samples representative of the set used to train the variational autoencoder. However, there is nothing encouraging the shape of the latent distribution. This is problematic in the sense of a basis — we wish to draw from a distribution and pass those drawn values to the decoder to generate new data samples, however, if we cannot be certain that we are capturing the entire variational autoencoder training space in our sampling the generated data will be biased.
Fortunately, there are two components to the variational autoencoder’s loss function: reconstruction loss and KL loss. The reconstruction loss is the same as in a standard autoencoder. The KL loss is responsible for encouraging the shape of the distribution in the latent space towards normality. This is extremely important, especially if we wish to generate new data samples — if the distribution in the latent space is normal, we can simply sample from the normal distribution and pass those values to the decoder to generate new inputs (this is a big deal).
Unfortunately, this can be where variational autoencoders become extremely difficult to work with. When developing the variational autoencoder for the volatility surface use case referenced above, it took two weeks to find the appropriate way to balance reconstruction and KL loss. If reconstruction error is prioritized, generated samples will be extremely poor but the reconstruction of inputs will thrive. If KL loss is prioritized, generated samples will be fantastic, but make literally no sense as the reconstruction of original inputs is completely forgone. That’s why it is absolutely crucial to strike a balance. If major efforts are going towards the construction of variational autoencoders for a specific use case I encourage you to read my paper as it exhibits in better detail (and provides references) to the procedure used for balancing these errors.
Again, there aren’t many implementations of variational autoencoders in Python. This can partially be attributed to the difficulty of implementation and lack of use cases. Nevertheless, I have provided the code below for dynamic implementation of variational autoencoders regardless of use case...
You’ll notice the variational encoder is using a log representation of standard deviation instead of standard deviation itself (which is also exhibited in the sample function, mathematically we are using standard deviation). This is to assist in the convergence of the KL loss term.
Just as before the variational encoder and variational decoder can be combined into a single model: a variational autoencoder.
Model: "VAE"_________________________________________________________________Layer (type)                 Output Shape              Param #   =================================================================encoder_input (InputLayer)   (None, 10)                0         _________________________________________________________________encoder (Model)              (None, 5)                 220       _________________________________________________________________decoder (Model)              (None, 10)                170       =================================================================Total params: 390Trainable params: 390Non-trainable params: 0_________________________________________________________________
The goal of this article was to give you an understanding of variational autoencoders with example code so you can develop your own generative machine learning model. We first looked at standard autoencoders from a background in simple feed-forward neural networks. The notion of a discrete latent space was then extended to a probabilistic latent space, the primary difference between autoencoders and variational autoencoders. Lastly, we considered some of the more technical aspects of the variational autoencoder and decomposed some of the issues with balancing optimization in the training space.
This is a particularly niche area of machine learning that I have spent a lot of time in for my last research project. If there are any questions regarding specific code or any of the models in general I’ll be happy to answer them.