Autoencoders: Neural Networks for Unsupervised Learning
By: Joseph Lee Wei En
Tags: Machine Learning, Deep Learning, Introduction, Unsupervised Learning, Autoencoder

Thus far, we’ve covered a very simplistic example; however, auto-encoders in practice are not far off in intuition. Recall that the blue and green arrows are simply functions that convert a large set of features into a smaller set of features and vice versa. Since neural networks are essentially complicated functions (recall Part 1a), we can use neural networks as the blue and green arrows! And that, in essence, is an auto-encoder!

An auto-encoder uses neural networks for the blue and green arrow that compresses and reconstructs the data respectively.

Just for terminology sake, we call the blue arrow the encoder and the green arrow the decoder. The smaller set of features that represent the data is called the encoding.

Now that we understand what the auto-encoder is trying to do, how do we train it? After all, we said at the start that labels are required to train our neural network! For this, we require some clever engineering based on some astute observations.

The first thing that we might have tried to do is to try to train the encoder and decoder as separate neural networks. However, we soon face the problem that we do not know the ground truth of what the encoding (compressed set of features) ought to be. There is no label to say that the input features should correspond to this particular encoding. Thus, it’s impossible to train our encoder! Without the encoder, we will not have the encoding and thus we have no input features to the decoder! This makes it impossible to train our decoder as well!

The problem of training a neural network for a blue and green arrow separately: we do not have the label for the encoder nor the input data for our decoder.

You might be wondering why I say that the decoder has labels in the table above. Where did these labels come from? After all, we do not have any external labels.

We must ask ourselves: What is the objective of the decoder? What is the ‘ground truth’ value that we are trying to output?

Let’s take an example. Suppose we wish to encode an image of a cat into some compressed set of features. The point of the decoder is to reconstruct the image of the cat. What is the ‘ground truth’ label for the decoder? The original image of the cat!

The ground truth label of the decoder is the original image of the cat, since the decoder is trying to reconstruct that image from the encoding.

Ok, so we have a label for the decoder. But that doesn’t solve our initial problem: we don’t have the label for the encoder and the input for the decoder.

To address this problem, we train the encoder and decoder as one large neural network rather than separately. The input features for our encoder is the input features for our large neural network. The label for our decoder is the label for our large neural network. This way, we have both the input features and label necessary to train our neural network:

From our above table, we saw that we had no label to the encoder and no input data to the decoder. By combining them, we get both the input data and label to our larger neural network.

It is not by accident that I called the encoding a “neuron” in the simplistic example I gave earlier. The output of the encoder is a set of neurons that forms the encoding (compressed set of features). The input of the decoder is the very same set of neurons in the encoding (compressed set of features). Since the input of a layer in the neural network is the output of the neurons in the previous layer, we can combine the encoder and decoder into a giant neural network like this:

A simplistic auto-encoder, where the three input features in blue are encoded into two features within the encoding, and then reconstructed to the three features in green.

Notice that while the encoder is on the left side and the decoder is on the right side, together they form one big neural network with three layers (blue, orange and green).

Of course, auto-encoders in practice have more layers in the encoder and decoder. In fact, they don’t even have to be fully-connected layers like we’ve shown above. Most image auto-encoders will have convolutional layers, and other layers we’ve seen in neural networks. The one thing they must have, however, is a bottleneck layer that corresponds to the encoding.

The point of the auto-encoder is to reduce the feature dimensions. If, in the above diagram, we had four orange neurons instead of two, then our encoding has more features than the input! This totally defeats the purpose of auto-encoders in the first place.

So now that we’ve got our large neural network architecture, how do we train it? Recall that the label of the decoder is now the label of this large neural network, and the label of the decoder was our original input data. Therefore, the label for our large neural network is exactly the same as the original input data to this large neural network!

For us to apply our neural networks and whatever we’ve learnt in Part 1a, we need to have a loss function that tells us how we are doing. We then find the best parameters that minimizes the loss function. This much has not changed. In other tasks, the loss function comes from how far away our output neuron is from the ground truth value. In this task, the loss function comes from how far away our output neuron is from our input neuron!

Given that the task is to encode and reconstruct, this makes intuitive sense. If the output neurons match the original data points perfectly, this means that we have successfully reconstructed the input. Since the neural network has a bottleneck layer, it must then mean that the fewer set of features in the encoding contains all the data it needs, which means we have a perfect encoder. This is the gold standard. Now, the auto-encoder may not be perfect, but the closer we can get to this gold standard, the better.

In essence, training an auto-encoder means:

Training a neural network with a ‘bottleneck layer’ within our neural network. The bottleneck layer has less features than the input layer. Everything to the left of the bottleneck layer is the encoder; everything to the right is the decoder.

The label that we compare our output against is the input to the neural network. Since we now have a label, we can apply our standard neural network training that we’ve learnt in Part 1a and Part 1b as though this was a Supervised Learning task.

And there we have it, our auto-encoder!

Summary: An auto-encoder uses a neural network for dimensionality reduction. This neural network has a bottleneck layer, which corresponds to the compressed vector. When we train this neural network, the ‘label’ of our output is our original input. Thus, the loss function we minimize corresponds to how poorly the original data is reconstructed from the compressed vector.