Autoencoder neural networks: what and how? | by Jake Krajewski | Towards Data Science

Preface
I’ll be walking through the creation of an autoencoder using Keras and Python. First, I’ll address what an autoencoder is and how would we possibly implement one. Then I’ll go through steps of actually creating one. Probably going to use MNIST because it’s generic and simple.
Introduction: What is an autoencoder?
Autoencoders take any input, chop it into some compressed version, and use that to reconstruct what the input was. So basically, input x goes into hidden layer h, h = f(x), and comes out as reconstruction r, r = g(h). The autoencoder is good when r is close to x, or when the output looks like the input.
So, is it a good thing to have a neural network that outputs exactly what the input was? In many cases, not really, but they’re often used for other purposes. One common objective is that the hidden layer h should have some limitations imposed on it such that it pulls out important details about x, without actually needing to keep all the information that x provided, thereby acting as a sort of lossy compression, and it should do this automatically from examples rather than being engineered by a human to recognize the salient features (Chollet, 2016).
Why tho?
Often when people write autoencoders, the hope is that the middle layer h will take on useful properties in some compressed format. Here we’ll be building something called an undercomplete autoencoder, which basically means we shrink the hidden layer so that there’s no way it can store the same amount of info that the input gives us, and from that, we try to get a nice reconstruction. Building a model like this forces the autoencoder to find the most salient features of the data you feed it. In real life, it can be used in reducing dimensionality of datasets, which can help for data visualization, or for potentially denoising noisy data.
Parts list
Here’s the basic list of things we’ll need to create.
input data— what is getting encoded and decoded?an encoding function — there needs to be a layer that takes an input and encodes it.a decoding function — there needs to be a layer that takes the encoded input and decodes it.loss function — The autoencoder is good when the output of the decoded version is very close to the original input data (loss is small), and bad when the decoded version looks nothing like the original input.
input data— what is getting encoded and decoded?
an encoding function — there needs to be a layer that takes an input and encodes it.
a decoding function — there needs to be a layer that takes the encoded input and decodes it.
loss function — The autoencoder is good when the output of the decoded version is very close to the original input data (loss is small), and bad when the decoded version looks nothing like the original input.
The Approach
The simplest autoencoder looks something like this: x → h → r, where the function f(x) results in h, and the function g(h) results in r. We’ll be using neural networks so we don’t need to calculate the actual functions.
Logically, step 1 will be to get some data. We’ll grab MNIST from the Keras dataset library. It’s comprised of 60,000 training examples and 10,000 test examples of handwritten digits 0–9. Next, we’ll do some basic data preparation so that we can feed it into our neural network as our input set, x.
Then in step 2, we’ll build the basic neural network model that gives us hidden layer h from x.
We’ll put together a single dense hidden layer that takes in x as input with a ReLU activation layer.Next, we’ll pass the output of this layer into another dense layer, and run the output through a sigmoid activation layer.
We’ll put together a single dense hidden layer that takes in x as input with a ReLU activation layer.
Next, we’ll pass the output of this layer into another dense layer, and run the output through a sigmoid activation layer.
Once we have a model, we’ll be able to train it in step 3, and then in step 4, we’ll visualize the output.
Let’s put it together:
First, let’s not forget the necessary imports to help us create our neural network (keras), do standard matrix mathematics (numpy), and plot our data (matplotlib). We’ll call this step 0.
# Importing modules to create our layers and model.from keras.layers import Input, Densefrom keras.models import Model# Importing standard utilsimport numpy as npimport matplotlib.pyplot as plt
Step 1. Import our data, and do some basic data preparation. Since we’re not going to use labels here, we only care about the x values.
from keras.datasets import mnist(train_xs, _), (test_xs, _) = mnist.load_data()
Next, we’ll normalize them between 0 and 1. Since they’re greyscale images, with values between 0 and 255, we’ll represent the input as float32's and divide by 255. This means if the value is 255, it’ll be normalized to 255.0/255.0 or 1.0, and so on and so forth.
# Note the '.' after the 255, this is correct for the type we're dealing with. It means do not interpret 255 as an integer. train_xs = train_xs.astype('float32') / 255.test_xs = test_xs.astype('float32') / 255.
Now think about this, we have images that are 28 x 28, with values between 0 and 1, and we want to pass them into a neural network layer as an input vector. What should we do? We could use a convolutional neural network, but in this simple case, we’ll just use a dense layer. So how do we feed it in? We’ll flatten each image into a single dimensional vector of 784 x 1 values (28 x 28 = 784).
train_xs = train_xs.reshape((len(train_xs), np.prod(train_xs.shape[1:])))test_xs = test_xs.reshape((len(test_xs), np.prod(test_xs.shape[1:])))
Step 2. Let’s put together a basic network. We’re simply going to create an encoding layer, and a decoding layer. We’ll put them together into a model called the autoencoder below. We’ll also decrease the size of the encoding so we can get some of that data compression. Here we’ll use 36 to keep it simple.
# Defining the level of compression of the hidden layer. Basically, as the input is passed through the encoding layer, it will come out smaller if you want it to find salient features. If I choose 784 for my encoding dimension, there would be a compression factor of 1, or nothing.encoding_dim = 36input_img = Input(shape=(784, ))# This is the size of the output. We want to generate 28 x 28 pictures in the end, so this is the size we're looking for. output_dim = 784encoded = Dense(encoding_dim, activation='relu')(input_img)decoded = Dense(output_dim, activation='sigmoid')(encoded)
Now create a model that accepts input_img as inputs and outputs the decoder layer. Then compile the model, in this case with adadelta as the optimizer and binary_crossentropy as the loss.
autoencoder = Model(input_img, decoded)autoencoder.compile(optimizer='adadelta', loss='binary_crossentropy')
Step 3. Our model is ready to train. You’ll be able to run this without a GPU, it doesn’t take long. We’ll call fit on the autoencoder model we created, passing in the x values for both the inputs and outputs, for 15 epochs, with a relatively large batch size (256). This will help it train somewhat quickly. We’ll enable shuffle to prevent homogeneous data in each batch and then we’ll use the test values as validation data. Note: if you want to train longer without over-fitting, sparseness and regularization may be added to your model.
autoencoder.fit(train_xs, train_xs, epochs=15, batch_size=256, shuffle=True, validation_data=(test_xs, test_xs)
That’s it. Autoencoder done. You’ll see it should have a loss of about 0.69 meaning that the reconstruction we’ve created generally represents the input fairly well from the highly compressed version. But can’t we take a look at it for ourselves?
Step 4. For this, we’ll do some inference to grab our reconstructions from our input data, and then we’ll display them with matplotlib. For this we want to use the predict method.
Here’s the thought process: take our test inputs, run them through autoencoder.predict, then show the originals and the reconstructions.
# Run your predictions and store them in a decoded_images list. decoded_images = autoencoder.predict(test_xs)
Here’s how you get that image above:
# We'll plot 10 images. n = 10plt.figure(figsize=(16, 3))for i in range(n):    # Show the originals    ax = plt.subplot(2, n, i + 1)    plt.imshow(test_xs[i].reshape(28, 28))    ax.get_xaxis().set_visible(False)    ax.get_yaxis().set_visible(False)# Show the reconstruction    ax = plt.subplot(2, n, i + 1 + n)    plt.imshow(decoded_imgs[i].reshape(28, 28))    ax.get_xaxis().set_visible(False)    ax.get_yaxis().set_visible(False)plt.show()
That’s it. That’s the most basic autoencoder. Many thanks to François Chollet, whose article I learned this from and inspired the basics of this tutorial, and Goodfellow, et al. for writing Deep Learning as an invaluable reference.
If you liked this tutorial or have suggestions, leave a comment below.
Edit: I’ve added the ability to view the hidden layer here which is definitely interesting. You can see that from these 6 x 6 images, the neural network is able to reconstruct the original input. Are you starting to see why this might be useful?
How would we get that middle row? Well that’s easy, we create a model out of the first hidden layer. Why? We do this so we can run the predict functionality and add its results to a list in python.
# Create a Model from the layer. encoder = Model(input_img, encoded)# Save the results to encoded_imgs. This must be done after the autoencoder model has been trained in order to use the trained weights.encoded_imgs = encoder.predict(test_xs)
Then we modify the matplotlib instructions a little bit to include the new images:
# We'll plot 10 images. n = 10plt.figure(figsize=(14, 4))for i in range(n):    # Show the originals    ax = plt.subplot(3, n, i + 1)    plt.imshow(test_xs[i].reshape(28, 28))    ax.get_xaxis().set_visible(False)    ax.get_yaxis().set_visible(False)    # Show the middle layer    ax = plt.subplot(3, n, i + 1 + n)    plt.imshow(encoded_imgs[i].reshape(6, 6))    ax.get_xaxis().set_visible(False)    ax.get_yaxis().set_visible(False)    # Show the reconstruction    ax = plt.subplot(3, n, i + 1 + n * 2)    plt.imshow(decoded_imgs[i].reshape(28, 28))    ax.get_xaxis().set_visible(False)    ax.get_yaxis().set_visible(False)plt.show()
That’s all for now! I hope this tutorial helped you understand a little about the thought processes behind autoencoders and how to use them in your neural networks.