Autoencoders and the Denoising Feature: From Theory to Practice
By: Lucas Robinet
Tags: Deep Learning, Editors Pick, TensorFlow, Autoencoder, Keras

Generalities about Autoencoders

If we had to summarize them in one sentence, it would probably sound like:

Autoencoders are neural network trained in an unsupervised way to attempt to copy inputs to outputs.

Yes I know, it may seem quite easy and useless. However, we will see that this is neither trivial nor pointless. In fact, Autoencoders are deep models capable of learning dense representations of the input. These representations are called latent representations or codings.

An Autoencoder has two distinct components :

An encoder : This part of the model takes in parameter the input data and compresses it. E(x) = c where x is the input data, c the latent representation and E our encoding function.

: This part of the model takes in parameter the input data and compresses it. E(x) = c where x is the input data, c the latent representation and E our encoding function. A decoder: This part takes in parameter the latent representation and try to reconstruct the original input. D(c) = x’ where x’ is the output of the decoder and D our decoding function

Undercomplete Autoencoders

Throughout the training phase, the goal is for our network to be able to learn how to reconstruct our input data. The following figure illustrates this idea by showing the Autoencoder model architecture.

However, most of the time, it is not the output of the decoder that interests us but rather the latent space representation. We hope that training the Autoencoder end-to-end will then allow our encoder to find useful features in our data.

The decoder, , is used to train the autoencoder end-to-end, but in practical applications, we often care more about the encoder and the codings.

To highlight important properties, one can, for example, constrain the latent space to be smaller than the dimension of the inputs. In this case, our model is an Undercomplete Autoencoders. In the majority of cases we work with this type of autoencoders since one of the main applications of this architecture is dimensionality reduction.

The learning process is quite regular, it aims at minimizing a loss function.

There are different metrics to quantify this loss function such as the Mean Square Error or the cross-entropy (when the activation function is a sigmoid for instance). This loss must penalize the reconstruction for being dissimilar from x.

Linear Autoencoders & Principal Component Analysis

So one of the main applications of Autoencoders is dimensionality reduction, just like a Principal Component Analysis (PCA). In fact, if the decoder is linear and the cost function is the Mean Square Error, an Autoencoder learns to span the same subspace as the PCA.

Comparing the two methods or explaining them in detail is outside the scope of this article. However, if you are interested, I recommend you to read this blog post (this one is also very interesting) which gives you a first intuition of PCA as well as these excellent slides which compare the two methods.

But if one were to summarize the two methods…

PCA: In PCA, we also seek to minimize the gap between input data and its reconstruction by measuring and minimizing the distance between the two. (For example the Euclidian distance.)

N.B: Vectors of the decoding matrix must have unit norm and be orthogonal

This optimization problem may be solved using Singular Value Decomposition. Specifically, the optimal P is given by the eigenvectors of the X covariance matrix corresponding to the largest eigenvalues. Again the full demonstration of that property is outside the scope of this tutorial.

Autoencoders: If we choose to train them with the Mean Square Error, then we aim at minimizing

In the case where f and g are linears, the loss function becomes

Both methods have the same objective function, which is convex, but uses two different ways to reach it. Indeed, Autoencoders are feedforward neural networks and are therefore trained as such with, for example, a Stochastic Gradient Descent.

In other words, the Optimal Solution of Linear Autoencoder is the PCA.

Now that the presentations are done, let’s look at how to use an autoencoder to do some dimensionality reduction.

For the sake of simplicity, we will simply project a 3-dimensional dataset into a 2-dimensional space. The first step to do such a task is to generate a 3D dataset. It is possible to do it simply with the following code.

And here is our dataset!

Now we have to create a Linear Autoencoder to perform PCA and this is possible with the following code.

And we can also visualize the corresponding graph!

Even if the code, as well as the graph, are quite self-explanatory, let’s take a closer look at them.

First, we define the encoder, a dense layer of 2 neurons that accepts inputs of dimension 3 (according to our dataset). So here we constrained the latent-space representation to be of dimension 2 (the output of the encoder).

Then, we define the decoder, also a dense layer but of 3 neurons this time because we want to reconstruct our 3-dimensional input at the output of the decoder.

The combination of the two forms the Autoencoder.

All that remains is to train it using data as inputs and targets. (We want to reconstruct our inputs remember ?)

And finally, the Autoencoder find the best 2D plane to project the data into while preserving as much variance as possible. Now let’s see the projection of our data!

And with the resulting plot…

So, the Autoencoder has reduced the dimensionality of our problem in the same way as PCA would have done.

Regularized Autoencoders

The problem is if we give our network too much capacity with many hidden layers, our model will be able to learn the task of copying data in inputs without extracting important information. In this case, the Autoencoder could not learn anything and would be overfitting. This phenomenon occurs as well with Undercomplete AE as Overcomplete AE (when the codings have higher dimensions than the inputs).

We want to be able to give our model capacity and not restrict it to just small networks to limit the number of parameters.

To do this, we use Regularized Autoencoders which encourages the model to develop new properties and to generalize better.

There are many different types of Regularized AE, but let’s review some interesting cases.

Sparse Autoencoders: it is simply an AE trained with a sparsity penalty added to his original loss function.

Sparse AEs are widespread for the classification task for instance. This sparsity penalty is simply a regularizer term added to a feedforward network.

Denoising Autoencoders: Adding noise (Gaussian for example) to the inputs forces our model to learn important features from our data.

Right now we are going to dive deeper into the concept of Denoising Autoencoders (DAE)