Title: Autoencoders with Keras, TensorFlow, and Deep Learning - PyImageSearch

URL Source: https://pyimagesearch.com/2020/02/17/autoencoders-with-keras-tensorflow-and-deep-learning/

Published Time: 2020-02-17T15:00:46+00:00

Markdown Content:
In this tutorial, you will learn how to implement and train autoencoders using Keras, TensorFlow, and Deep Learning.

[![Image 1](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20600%20400'%3E%3C/svg%3E)](https://pyimagesearch.com/wp-content/uploads/2020/02/keras_autoencoders_header.png)

Today’s tutorial kicks off a three-part series on the applications of autoencoders:

1.  _Autoencoders with Keras, TensorFlow, and Deep Learning_ (today’s tutorial)
2.  _Denoising autoenecoders with Keras and TensorFlow_ (next week’s tutorial)
3.  _Anomaly detection with Keras, TensorFlow, and Deep Learning_ (tutorial two weeks from now)

A few weeks ago, I published an [introductory guide to anomaly/outlier detection](https://pyimagesearch.com/2020/01/20/intro-to-anomaly-detection-with-opencv-computer-vision-and-scikit-learn/) using _standard_ machine learning algorithms.

My intention was to _immediately_ follow up that post with a a guide on _deep learning-based_ anomaly detection; however, as I started writing the code for the tutorial, _I realized I had never covered autoencoders on the PyImageSearch blog!_

Trying to discuss deep learning-based anomaly detection without prior context on _what_ autoencoders are and _how_ they work would be challenging to follow, comprehend, and digest.

Therefore, we’re going to spend the next couple of weeks looking at autoencoder algorithms, including their practical, real-world applications.

**To learn about the fundamentals of autoencoders using Keras and TensorFlow, _just keep reading!_**

![Image 2](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20128%20128'%3E%3C/svg%3E)

#### Looking for the source code to this post?

[Jump Right To The Downloads Section](#download-the-code)

In the first part of this tutorial, we’ll discuss what **autoencoders** are, including how **convolutional autoencoders** can be applied to image data. We’ll also discuss the _difference_ between autoencoders and other generative models, such as **Generative Adversarial Networks (GANs).**

From there, I’ll show you how to implement and train a convolutional autoencoder using Keras and TensorFlow.

We’ll then review the results of the training script, including visualizing how the autoencoder did at reconstructing the input data.

Finally, I’ll recommend next steps to you if you are interested in learning more about deep learning applied to image datasets.

### What are autoencoders?

Autoencoders are a type of **unsupervised neural network** (i.e., no class labels or labeled data) that seek to:

1.  Accept an input set of data (i.e., the _input_).
2.  Internally _compress_ the input data into a **latent-space representation** (i.e., a single vector that _compresses_ and _quantifies_ the input).
3.  **Reconstruct the input data** from this latent representation (i.e., the output).

Typically, we think of an autoencoder having two components/subnetworks:

1.  **Encoder:** Accepts the input data and compresses it into the latent-space. If we denote our input data as ![Image 3: x](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%209%207'%3E%3C/svg%3E) and the encoder as ![Image 4: E](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%2013%2011'%3E%3C/svg%3E), then the output latent-space representation, ![Image 5: s](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%207%207'%3E%3C/svg%3E), would be ![Image 6: s = E(x)](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%2062%2018'%3E%3C/svg%3E).
2.  **Decoder:** The decoder is responsible for accepting the latent-space representation ![Image 7: s](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%207%207'%3E%3C/svg%3E) and then reconstructing the original input. If we denote the decoder function as ![Image 8: D](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%2014%2011'%3E%3C/svg%3E) and the output of the detector as ![Image 9: o](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%208%207'%3E%3C/svg%3E), then we can represent the decoder as ![Image 10: o = D(s)](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%2063%2018'%3E%3C/svg%3E).

Using our mathematical notation, the entire training process of the autoencoder can be written as:

![Image 11: o = D(E(x))](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%2089%2018'%3E%3C/svg%3E)

**Figure 1** below demonstrates the basic architecture of an autoencoder:

[![Image 12](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20600%20250'%3E%3C/svg%3E)](https://pyimagesearch.com/wp-content/uploads/2020/02/keras_autoencoder_arch_flow.png)

**Figure 1:** Autoencoders with Keras, TensorFlow, Python, and Deep Learning don’t have to be complex. Breaking the concept down to its parts, you’ll have an input image that is passed through the autoencoder which results in a similar output image. (figure inspired by Nathan Hubens’ article, [_Deep inside: Autoencoders_](https://towardsdatascience.com/deep-inside-autoencoders-7e41f319999f))

Here you can see that:

1.  We input a digit to the autoencoder.
2.  The encoder subnetwork creates a latent representation of the digit. This latent representation is **_substantially smaller_** (in terms of dimensionality) than the input.
3.  The decoder subnetwork then reconstructs the original digit from the latent representation.

**You can thus think of an autoencoder as a network that _reconstructs its input!_**

To train an autoencoder, we input our data, attempt to reconstruct it, and then minimize the mean squared error (or similar loss function).

**Ideally, the output of the autoencoder will be _near identical_ to the input.**

### An autoencoder reconstructs it’s input — _so what’s the big deal?_

[![Image 13](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20600%20316'%3E%3C/svg%3E)](https://pyimagesearch.com/wp-content/uploads/2020/02/keras_autoencoder_whats_the_point.png)

**Figure 2:** Autoencoders are useful for compression, dimensionality reduction, denoising, and anomaly/outlier detection. In this tutorial, we’ll use Python and Keras/TensorFlow to train a deep learning autoencoder. ([image source](https://lilianweng.github.io/lil-log/2018/08/12/from-autoencoder-to-beta-vae.html))

At this point, some of you might be thinking:

> Adrian, what’s the big deal here?
> 
> If the goal of an autoencoder is just to _reconstruct_ the input, why even use the network in the first place?
> 
> If I wanted a _copy_ of my input data, I could literally just copy it with a single function call.
> 
> Why on earth would I apply deep learning and go through the trouble of training a network?

This question, although a legitimate one, does indeed contain a large misconception regarding autoencoders.

Yes, during the training process, our goal is to train a network that can learn how to reconstruct our input data — **but the true value of the autoencoder lives inside that latent-space representation.**

Keep in mind that autoencoders _compress_ our input data and, more to the point, when we train autoencoders, what we _really_ care about is the encoder, ![Image 14: E](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%2013%2011'%3E%3C/svg%3E), and the latent-space representation, ![Image 15: s = E(x)](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%2062%2018'%3E%3C/svg%3E)_._

The decoder, ![Image 16: o = D(s)](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%2063%2018'%3E%3C/svg%3E), is used to train the autoencoder end-to-end, but in practical applications, we often (but not always) care more about the encoder and the latent-space.

Later in this tutorial, we’ll be training an autoencoder on the MNIST dataset. The MNIST dataset consists of digits that are _28×28_ pixels with a single channel, implying that each digit is represented by _28 x 28 = 784_ values. **The autoencoder we’ll be training here will be able to compress those digits into a vector of only _16 values_ — that’s a reduction of _nearly 98%!_**

So what can we do if an input data point is compressed into such a small vector?

That’s where things get really interesting.

### What are applications of autoencoders?

[![Image 17](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20600%20233'%3E%3C/svg%3E)](https://pyimagesearch.com/wp-content/uploads/2020/02/keras_autoencoders_applications.png)

**Figure 3:** Autoencoders are typically used for dimensionality reduction, denoising, and anomaly/outlier detection. Outside of computer vision, they are extremely useful for Natural Language Processing (NLP) and text comprehension. In this tutorial, we’ll use Python and Keras/TensorFlow to train a deep learning autoencoder. ([image source](https://towardsdatascience.com/deep-autoencoders-using-tensorflow-c68f075fd1a3))

Autoencoders are typically used for:

*   **Dimensionality reduction** (i.e., think PCA but more powerful/intelligent).
*   **Denoising** (ex., removing noise and preprocessing images to improve OCR accuracy).
*   **Anomaly/outlier detection** (ex., detecting mislabeled data points in a dataset or detecting when an input data point falls well outside our typical data distribution).

Outside of the computer vision field, you’ll see autoencoders applied to Natural Language Processing (NLP) and text comprehension problems, including understanding the semantic meaning of words, constructing word embeddings, and even text summarization.

### How are autoencoders different from GANs?

If you’ve done any prior work with Generative Adversarial Networks (GANs), you might be wondering how autoencoders are different from GANs.

Both GANs and autoencoders are **generative models**; however, an autoencoder is essentially learning an _identity function_ via _compression._

The autoencoder will accept our input data, compress it down to the latent-space representation, and then attempt to reconstruct the input using just the latent-space vector.

Typically, the latent-space representation will have **_much fewer dimensions_** than the original input data.

**GANs on the other hand:**

1.  Accept a low dimensional input.
2.  Build a high dimensional space from it.
3.  Generate the final output, which is _not_ part of the original training data but ideally _passes_ as such.

Furthermore, GANs have an **_evolving loss landscape,_** which autoencoders do not.

As a GAN is trained, the **generative model** generates “fake” images that are then mixed with actual “real” images — the **discriminator model** must then determine which images are “real” vs. “fake/generated”.

As the generative model becomes better and better at generating fake images that can fool the discriminator, the loss landscape evolves and changes (this is one of the reasons why training GANs is so damn hard).

While both GANs and autoencoders are generative models, most of their similarities end there.

Autoencoders cannot generate new, realistic data points that could be considered “passable” by humans. Instead, autoencoders are primarily used as a method to compress input data points into a latent-space representation. That latent-space representation can then be used for compression, denoising, anomaly detection, etc.

For more details on the differences between GANs and autoencoders, [I suggest giving this thread on Quora a read](https://www.quora.com/What-is-the-difference-between-Generative-Adversarial-Networks-and-Autoencoders).

### Configuring your development environment

To follow along with today’s tutorial on autoencoders, you should use TensorFlow 2.0. I have two installation tutorials for TF 2.0 and associated packages to bring your development system up to speed:

*   _[How to install TensorFlow 2.0 on Ubuntu](https://pyimagesearch.com/2019/12/09/how-to-install-tensorflow-2-0-on-ubuntu/)_ (Ubuntu 18.04 OS; CPU and optional NVIDIA GPU)
*   _[How to install TensorFlow 2.0 on macOS](https://pyimagesearch.com/2019/12/09/how-to-install-tensorflow-2-0-on-macos/)_ (Catalina and Mojave OSes)

**Please note:** PyImageSearch does not support Windows — [refer to our FAQ](https://pyimagesearch.com/faqs/single-faq/can-you-help-me-do-___-on-windows).

### Project structure

Be sure to grab the **_“Downloads”_** associated with the blog post. From there, extract the .zip and inspect the file/folder layout:

$ tree --dirsfirst
.
├── pyimagesearch
│   ├── \_\_init\_\_.py
│   └── convautoencoder.py
├── output.png
├── plot.png
└── train\_conv\_autoencoder.py

1 directory, 5 files

We will review two Python scripts today:

*   `convautoencoder.py`: Contains the `ConvAutoencoder` class and `build` method required to assemble our neural network with `tf.keras`.
*   `train_conv_autoencoder.py`: Trains a digits autoencoder on the MNIST dataset. Once the autoencoder is trained, we’ll loop over a number of output examples and write them to disk for later inspection.

Our training script results in both a `plot.png` figure and `output.png` image. The output image contains side-by-side samples of the original versus reconstructed image.

In the next section, we will implement our autoencoder with the high-level Keras API built into TensorFlow.

### Implementing a convolutional autoencoder with Keras and TensorFlow

Before we can train an autoencoder, we first need to implement the autoencoder architecture itself.

**To do so, we’ll be using Keras and TensorFlow.**

My implementation loosely [follows Francois Chollet’s own implementation of autoencoders](https://blog.keras.io/building-autoencoders-in-keras.html) on the official Keras blog. My primary contribution here is to go into a bit more detail regarding the implementation itself.

Open up the `convautoencoder.py` file in your project structure, and insert the following code:

\# import the necessary packages
from tensorflow.keras.layers import BatchNormalization
from tensorflow.keras.layers import Conv2D
from tensorflow.keras.layers import Conv2DTranspose
from tensorflow.keras.layers import LeakyReLU
from tensorflow.keras.layers import Activation
from tensorflow.keras.layers import Flatten
from tensorflow.keras.layers import Dense
from tensorflow.keras.layers import Reshape
from tensorflow.keras.layers import Input
from tensorflow.keras.models import Model
from tensorflow.keras import backend as K
import numpy as np

class ConvAutoencoder:
	@staticmethod
	def build(width, height, depth, filters=(32, 64), latentDim=16):
		# initialize the input shape to be "channels last" along with
		# the channels dimension itself
		# channels dimension itself
		inputShape = (height, width, depth)
		chanDim = -1

We begin with a selection of imports from `tf.keras` and one from NumPy. If you don’t have TensorFlow 2.0 installed on your system, refer to the _“Configuring your development environment”_ section above.

Our `ConvAutoencoder` class contains one static method, `build`, which accepts five parameters:

*   `width`: Width of the input image in pixels.
*   `height`: Height of the input image in pixels.
*   `depth`: Number of channels (i.e., depth) of the input volume.
*   `filters`: A tuple that contains the set of filters for convolution operations. By default, this parameter includes both `32` and `64` filters.
*   `latentDim`: The number of neurons in our fully-connected (`Dense`) latent vector. By default, if this parameter is not passed, the value is set to `16`.

From there, we initialize the `inputShape` and channel dimension (we assume “channels last” ordering).

We’re now ready to initialize our input and begin adding layers to our network:

		# define the input to the encoder
		inputs = Input(shape=inputShape)
		x = inputs

		# loop over the number of filters
		for f in filters:
			# apply a CONV => RELU => BN operation
			x = Conv2D(f, (3, 3), strides=2, padding="same")(x)
			x = LeakyReLU(alpha=0.2)(x)
			x = BatchNormalization(axis=chanDim)(x)

		# flatten the network and then construct our latent vector
		volumeSize = K.int\_shape(x)
		x = Flatten()(x)
		latent = Dense(latentDim)(x)

		# build the encoder model
		encoder = Model(inputs, latent, name="encoder")

**Lines 25 and 26** define the input to the encoder.

With our inputs ready, we go loop over the number of `filters` and add our sets of `CONV=>LeakyReLU=>BN` layers (**Lines 29-33**).

Next, we flatten the network and construct our latent vector (**Lines 36-38**) — **this is our actual latent-space representation (i.e., the “compressed” data representation).**

We then build our `encoder` model (**Line 41**).

If we were to do a `print(encoder.summary())` of the `encoder`, assuming _28×28_ single channel images (`depth=1`) and `filters=(32, 64)` and `latentDim=16`, we would have the following:

Model: "encoder"
\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_
Layer (type)                 Output Shape              Param #
=================================================================
input\_1 (InputLayer)         \[(None, 28, 28, 1)\]       0
\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_
conv2d (Conv2D)              (None, 14, 14, 32)        320
\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_
leaky\_re\_lu (LeakyReLU)      (None, 14, 14, 32)        0
\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_
batch\_normalization (BatchNo (None, 14, 14, 32)        128
\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_
conv2d\_1 (Conv2D)            (None, 7, 7, 64)          18496
\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_
leaky\_re\_lu\_1 (LeakyReLU)    (None, 7, 7, 64)          0
\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_
batch\_normalization\_1 (Batch (None, 7, 7, 64)          256
\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_
flatten (Flatten)            (None, 3136)              0
\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_
dense (Dense)                (None, 16)                50192
=================================================================
Total params: 69,392
Trainable params: 69,200
Non-trainable params: 192
\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_

Here we can observe that:

*   Our encoder begins by accepting a _28x28x1_ input volume.
*   We then apply two rounds of `CONV=>RELU=>BN`, each with _3×3_ strided convolution. The strided convolution allows us to reduce the spatial dimensions of our volumes.
*   After applying our final batch normalization, we end up with a _7x7x64_ volume, which is flattened into a _3136_\-dim vector.
*   Our fully-connected layer (i.e., the `Dense` layer) serves our as our **latent-space representation.**

Next, let’s learn how the decoder model can take this latent-space representation and reconstruct the original input image:

		# start building the decoder model which will accept the
		# output of the encoder as its inputs
		latentInputs = Input(shape=(latentDim,))
		x = Dense(np.prod(volumeSize\[1:\]))(latentInputs)
		x = Reshape((volumeSize\[1\], volumeSize\[2\], volumeSize\[3\]))(x)

		# loop over our number of filters again, but this time in
		# reverse order
		for f in filters\[::-1\]:
			# apply a CONV\_TRANSPOSE => RELU => BN operation
			x = Conv2DTranspose(f, (3, 3), strides=2,
				padding="same")(x)
			x = LeakyReLU(alpha=0.2)(x)
			x = BatchNormalization(axis=chanDim)(x)

To start building the decoder model, we:

*   Construct the input to the decoder model based on the `latentDim`. (**Lines 45 and 46**).
*   Accept the 1D `latentDim` vector and turn it into a 2D volume so that we can start applying convolution (**Line 47**).
*   Loop over the number of filters, this time in reverse order while applying a `CONV_TRANSPOSE => RELU => BN` operation (**Lines 51-56**).

Transposed convolution is used to increase the _spatial dimensions_ (i.e., width and height) of the volume.

Let’s finish creating our autoencoder:

		# apply a single CONV\_TRANSPOSE layer used to recover the
		# original depth of the image
		x = Conv2DTranspose(depth, (3, 3), padding="same")(x)
		outputs = Activation("sigmoid")(x)

		# build the decoder model
		decoder = Model(latentInputs, outputs, name="decoder")

		# our autoencoder is the encoder + decoder
		autoencoder = Model(inputs, decoder(encoder(inputs)),
			name="autoencoder")

		# return a 3-tuple of the encoder, decoder, and autoencoder
		return (encoder, decoder, autoencoder)

Wrapping up, we:

*   Apply a final `CONV_TRANSPOSE` layer used to recover the original channel depth of the image (1 channel for single channel/grayscale images or 3 channels for RGB images) on **Line 60**.
*   Apply a sigmoid activation function (**Line 61**).
*   Build the `decoder` model, and add it with the `encoder` to the `autoencoder` (**Lines 64-68**). **The autoencoder becomes the encoder + decoder.**
*   Return a 3-tuple of the encoder, decoder, and autoencoder.

If we were to complete a `print(decoder.summary())` operation here, we would have the following:

Model: "decoder"
\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_
Layer (type)                 Output Shape              Param #
=================================================================
input\_2 (InputLayer)         \[(None, 16)\]              0
\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_
dense\_1 (Dense)              (None, 3136)              53312
\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_
reshape (Reshape)            (None, 7, 7, 64)          0
\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_
conv2d\_transpose (Conv2DTran (None, 14, 14, 64)        36928
\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_
leaky\_re\_lu\_2 (LeakyReLU)    (None, 14, 14, 64)        0
\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_
batch\_normalization\_2 (Batch (None, 14, 14, 64)        256
\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_
conv2d\_transpose\_1 (Conv2DTr (None, 28, 28, 32)        18464
\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_
leaky\_re\_lu\_3 (LeakyReLU)    (None, 28, 28, 32)        0
\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_
batch\_normalization\_3 (Batch (None, 28, 28, 32)        128
\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_
conv2d\_transpose\_2 (Conv2DTr (None, 28, 28, 1)         289
\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_
activation (Activation)      (None, 28, 28, 1)         0
=================================================================
Total params: 109,377
Trainable params: 109,185
Non-trainable params: 192
\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_

The `decoder` accepts our 16-dim latent representation from the `encoder` and then builds a _new_ fully-connected layer of 3136-dim, which is the product of _7 x 7 x 64 = 3136._

Using our new 3136-dim FC layer, we reshape it into a 3D volume of _7 x 7 x 64._ From there we can start applying our `CONV_TRANSPOSE=>RELU=>BN` operation. Unlike standard strided convolution, which is used to _decrease_ volume size, our transposed convolution is used to _increase_ volume size.

Finally, a transposed convolution layer is applied to recover the **original channel depth** of the image. Since our images are grayscale, we learn a _single filter_, the output of which is a _28 x 28 x 1_ volume (i.e., the dimensions of the original MNIST digit images).

A `print(autoencoder.summary())` operation shows the composed nature of the encoder and decoder:

Model: "autoencoder"
\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_
Layer (type)                 Output Shape              Param #
=================================================================
input\_1 (InputLayer)         \[(None, 28, 28, 1)\]       0
\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_
encoder (Model)              (None, 16)                69392
\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_
decoder (Model)              (None, 28, 28, 1)         109377
=================================================================
Total params: 178,769
Trainable params: 178,385
Non-trainable params: 384
\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_

The input to our `encoder` is the **original** _28 x 28 x 1_ images from the MNIST dataset. Our `encoder` then learns a 16-dim latent-space representation of the data, after which the `decoder` **reconstructs** the original _28 x 28 x 1_ images.

In the next section, we will develop our script to train our autoencoder.

### Creating the convolutional autoencoder training script

With our autoencoder architecture implemented, let’s move on to the training script.

Open up the `train_conv_autoencoder.py` in your project directory structure, and insert the following code:

\# set the matplotlib backend so figures can be saved in the background
import matplotlib
matplotlib.use("Agg")

# import the necessary packages
from pyimagesearch.convautoencoder import ConvAutoencoder
from tensorflow.keras.optimizers import Adam
from tensorflow.keras.datasets import mnist
import matplotlib.pyplot as plt
import numpy as np
import argparse
import cv2

# construct the argument parse and parse the arguments
ap = argparse.ArgumentParser()
ap.add\_argument("-s", "--samples", type=int, default=8,
	help="# number of samples to visualize when decoding")
ap.add\_argument("-o", "--output", type=str, default="output.png",
	help="path to output visualization file")
ap.add\_argument("-p", "--plot", type=str, default="plot.png",
	help="path to output plot file")
args = vars(ap.parse\_args())

On **Lines 2-12,** we handle our imports. We’ll use the `"Agg"` backend of `matplotlib` so that we can export our training plot to disk.

We need our custom `ConvAutoencoder` architecture class which we implemented in the previous section.

We will use the `Adam` optimizer as we train on the MNIST benchmarking dataset. For visualization, we’ll employ OpenCV.

Next, we’ll parse three [command line arguments](https://pyimagesearch.com/2018/03/12/python-argparse-command-line-arguments/), all of which are optional:

*   `--samples`: The number of output samples for visualization. By default this value is set to `8`.
*   `--output`: The path the output visualization image. We’ll name our visualization output.png by default
*   `--plot`: The path to our matplotlib output plot. A default of plot.png is assigned if this argument is not provided in the terminal.

Now we’ll set a couple hyperparameters and preprocess our MNIST dataset:

\# initialize the number of epochs to train for and batch size
EPOCHS = 25
BS = 32

# load the MNIST dataset
print("\[INFO\] loading MNIST dataset...")
((trainX, \_), (testX, \_)) = mnist.load\_data()

# add a channel dimension to every image in the dataset, then scale
# the pixel intensities to the range \[0, 1\]
trainX = np.expand\_dims(trainX, axis=-1)
testX = np.expand\_dims(testX, axis=-1)
trainX = trainX.astype("float32") / 255.0
testX = testX.astype("float32") / 255.0

**Lines 25 and 26** initialize the batch size and number of training epochs.

From there, we’ll work with our MNIST dataset. TensorFlow/Keras has a handy `load_data` method that we can call on `mnist` to grab the data (**Line 30**). From there, **Lines 34-37** (1) add a channel dimension to every image in the dataset and (2) scale the pixel intensities to the range _\[0, 1\]_.

We’re now ready to **build and train our autoencoder:**

\# construct our convolutional autoencoder
print("\[INFO\] building autoencoder...")
(encoder, decoder, autoencoder) = ConvAutoencoder.build(28, 28, 1)
opt = Adam(lr=1e-3)
autoencoder.compile(loss="mse", optimizer=opt)

# train the convolutional autoencoder
H = autoencoder.fit(
	trainX, trainX,
	validation\_data=(testX, testX),
	epochs=EPOCHS,
	batch\_size=BS)

To build the convolutional autoencoder, we call the `build` method on our `ConvAutoencoder` class and pass the necessary arguments (**Line 41**). Recall that this results in the `(encoder, decoder, autoencoder)` tuple — going forward in this script, we only need the `autoencoder` for training and predictions.

We initialize our `Adam` optimizer with an initial learning rate of `1e-3` and go ahead and compile it with mean-squared error loss (**Lines 42 and 43**).

From there, we `fit` (train) our `autoencoder` on the MNIST data (**Lines 46-50**).

Let’s go ahead and plot our training history:

\# construct a plot that plots and saves the training history
N = np.arange(0, EPOCHS)
plt.style.use("ggplot")
plt.figure()
plt.plot(N, H.history\["loss"\], label="train\_loss")
plt.plot(N, H.history\["val\_loss"\], label="val\_loss")
plt.title("Training Loss and Accuracy")
plt.xlabel("Epoch #")
plt.ylabel("Loss/Accuracy")
plt.legend(loc="lower left")
plt.savefig(args\["plot"\])

And from there, we’ll make predictions on our testing set:

\# use the convolutional autoencoder to make predictions on the
# testing images, then initialize our list of output images
print("\[INFO\] making predictions...")
decoded = autoencoder.predict(testX)
outputs = None

# loop over our number of output samples
for i in range(0, args\["samples"\]):
	# grab the original image and reconstructed image
	original = (testX\[i\] \* 255).astype("uint8")
	recon = (decoded\[i\] \* 255).astype("uint8")

	# stack the original and reconstructed image side-by-side
	output = np.hstack(\[original, recon\])

	# if the outputs array is empty, initialize it as the current
	# side-by-side image display
	if outputs is None:
		outputs = output

	# otherwise, vertically stack the outputs
	else:
		outputs = np.vstack(\[outputs, output\])

# save the outputs image to disk
cv2.imwrite(args\["output"\], outputs)

**Line 67** makes predictions on the test set. We then loop over the number of `--samples` passed as a command line argument (**Line 71**) so that we can build our visualization. Inside the loop, we:

*   Grab _both_ the original and reconstructed images (**Lines 73 and 74**).
*   Stack the pair of images _side-by-side_ (**Line 77**).
*   Stack the pairs _vertically_ (**Lines 81-86**).
*   Finally, we output the visualization image to disk (**Line 89**).

In the next section, we’ll see the results of our hard work.

### Training the convolutional autoencoder with Keras and TensorFlow

We are now ready to see our autoencoder in action!

Make sure you use the **_“Downloads”_** section of this post to download the source code — from there you can execute the following command:

$ python train\_conv\_autoencoder.py
\[INFO\] loading MNIST dataset...
\[INFO\] building autoencoder...
Train on 60000 samples, validate on 10000 samples
Epoch 1/25
60000/60000 \[==============================\] - 68s 1ms/sample - loss: 0.0188 - val\_loss: 0.0108
Epoch 2/25
60000/60000 \[==============================\] - 68s 1ms/sample - loss: 0.0104 - val\_loss: 0.0096
Epoch 3/25
60000/60000 \[==============================\] - 68s 1ms/sample - loss: 0.0094 - val\_loss: 0.0086
Epoch 4/25
60000/60000 \[==============================\] - 68s 1ms/sample - loss: 0.0088 - val\_loss: 0.0086
Epoch 5/25
60000/60000 \[==============================\] - 68s 1ms/sample - loss: 0.0084 - val\_loss: 0.0080
...
Epoch 20/25
60000/60000 \[==============================\] - 83s 1ms/sample - loss: 0.0067 - val\_loss: 0.0069
Epoch 21/25
60000/60000 \[==============================\] - 83s 1ms/sample - loss: 0.0066 - val\_loss: 0.0069
Epoch 22/25
60000/60000 \[==============================\] - 83s 1ms/sample - loss: 0.0066 - val\_loss: 0.0068
Epoch 23/25
60000/60000 \[==============================\] - 83s 1ms/sample - loss: 0.0066 - val\_loss: 0.0068
Epoch 24/25
60000/60000 \[==============================\] - 83s 1ms/sample - loss: 0.0065 - val\_loss: 0.0067
Epoch 25/25
60000/60000 \[==============================\] - 83s 1ms/sample - loss: 0.0065 - val\_loss: 0.0068
\[INFO\] making predictions...

[![Image 18](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20600%20450'%3E%3C/svg%3E)](https://pyimagesearch.com/wp-content/uploads/2020/02/plot.png)

**Figure 4:** Our deep learning autoencoder training history plot was generated with matplotlib. Our autoencoder was trained with Keras, TensorFlow, and Deep Learning.

As **Figure 4** and the terminal output demonstrate, our training process was able to minimize the reconstruction loss of the autoencoder.

**But how well did the autoencoder do at reconstructing the training data?**

The answer is **_very good:_**

[![Image 19](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20327%20425'%3E%3C/svg%3E)](https://pyimagesearch.com/wp-content/uploads/2020/02/keras_autoencoder_output.png)

**Figure 5:** A sample of of Keras/TensorFlow deep learning autoencoder inputs _(left)_ and outputs _(right)_.

In **Figure 5**, on the _left_ is our original image while the _right_ is the reconstructed digit predicted by the autoencoder. **As you can see, the digits are _nearly indistinguishable_ from each other!**

At this point, you may be thinking:

> Great … so I can train a network to reconstruct my original image.
> 
> But you said that what _really matters_ is the internal latent-space representation.
> 
> How can I access that representation, and how can I use it for denoising and anomaly/outlier detection?

Those are great questions — **I’ll be addressing both in my next two tutorials here on PyImageSearch, so stay tuned!**

### What's next? We recommend [PyImageSearch University](https://pyimagesearch.com/pyimagesearch-university/?utm_source=blogPost&utm_medium=bottomBanner&utm_campaign=What%27s%20next%3F%20I%20recommend).

**Course information:**  
84 total classes • 114+ hours of on-demand code walkthrough videos • Last updated: February 2024  
★★★★★ 4.84 (128 Ratings) • 16,000+ Students Enrolled

**I strongly believe that if you had the right teacher you could _master_ computer vision and deep learning.**

Do you think learning computer vision and deep learning has to be time-consuming, overwhelming, and complicated? Or has to involve complex mathematics and equations? Or requires a degree in computer science?

That’s _not_ the case.

All you need to master computer vision and deep learning is for someone to explain things to you in _simple, intuitive_ terms. _And that’s exactly what I do_. My mission is to change education and how complex Artificial Intelligence topics are taught.

If you're serious about learning computer vision, your next stop should be PyImageSearch University, the most comprehensive computer vision, deep learning, and OpenCV course online today. Here you’ll learn how to _successfully_ and _confidently_ apply computer vision to your work, research, and projects. Join me in computer vision mastery.

**Inside PyImageSearch University you'll find:**

*   ✓ **84 courses** on essential computer vision, deep learning, and OpenCV topics
*   ✓ **84 Certificates** of Completion
*   ✓ **114+ hours** of on-demand video
*   ✓ **Brand new courses released _regularly_**, ensuring you can keep up with state-of-the-art techniques
*   ✓ **Pre-configured Jupyter Notebooks in Google Colab**
*   ✓ Run all code examples in your web browser — works on Windows, macOS, and Linux (no dev environment configuration required!)
*   ✓ Access to **centralized code repos for _all_ 536+ tutorials** on PyImageSearch
*   ✓ **Easy one-click downloads** for code, datasets, pre-trained models, etc.
*   ✓ **Access** on mobile, laptop, desktop, etc.

[Click here to join PyImageSearch University](https://pyimagesearch.com/pyimagesearch-university/?utm_source=blogPost&utm_medium=bottomBanner&utm_campaign=What%27s%20next%3F%20I%20recommend)

Summary
-------

In this tutorial, you learned the fundamentals of autoencoders.

Autoencoders are generative models that consist of an _encoder_ and a _decoder_ model. When trained, the encoder takes input data point and learns a latent-space representation of the data. This latent-space representation is a _compressed_ representation of the data, allowing the model to represent it in _far fewer_ parameters than the original data.

The decoder model then takes the latent-space representation and attempts to reconstruct the original data point from it. When trained end-to-end, the encoder and decoder function in a composed manner.

In practice, we use autoencoders for dimensionality reduction, compression, denoising, and anomaly detection.

After we understood the fundamentals, we implemented a convolutional autoencoder using Keras and TensorFlow.

In next week’s tutorial, we’ll learn how to use a convolutional autoencoder for denoising.

**To download the source code to this post (and be notified when future tutorials are published here on PyImageSearch), _just enter your email address in the form below!_**

![Image 20](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20410%20567'%3E%3C/svg%3E)

#### Download the Source Code and FREE 17-page Resource Guide

Enter your email address below to get a .zip of the code and a **FREE 17-page Resource Guide on Computer Vision, OpenCV, and Deep Learning.** Inside you'll find my hand-picked tutorials, books, courses, and libraries to help you master CV and DL!