Machine Learning: Autoencoders. Using autoencoders to fit high... | by Anuradha Wickramarachchi | Towards Data Science

I found the simplest definition for an autoencoder through Wikipedia, which translates itself into “A machine learning model that learns a lower-dimensional encoding of data”. This is one of the smartest ways of reducing the dimensionality of a dataset, just by using the capabilities of the differentiation ending (Tensorflow, PyTorch, etc). Indeed, we should have a specific neural network architecture in order to achieve this. However, before we start, let’s have a brief overview of dimensionality reduction.
Both PCA and Autoencoders intend to learn a lower-dimensional representation thus reducing the reconstruction error of original dimensionality by the lower dimensional representations. However, as you might know, PCA translates the original data to a lower set of dimensions that are orthogonal to each other. We call this a linear transformation into another space. The problem comes with the linear part. The information loss could be higher. In contrast with Autoencoders, the neural networks use gradient descent to estimate the best possible parameters for a hidden layer with lower dimensions.
Autoencoders take an architecture of a stream of data passing through a bottleneck. This bottleneck represents the lower dimension. The autoencoder is expected to learn this lower dimension thus minimizing the error score defined between the input and the output.
Let us have a look at a simple autoencoder (from Wikipedia).
As you can see we have a middle element which is a narrow passage. Input and Output for the training are usually the same since we intend to learn a lower-dimensional representation.
Our usual imports will be;
import tensorflow as tfimport numpy as npimport matplotlib.pyplot as plt # for plots
Processing data; taking MNIST handwriting dataset as an example.
(x_train, y_train), (x_test, y_test) = tf.keras.datasets.mnist.load_data()x_train = x_train.astype('float32') / 255x_test = x_test.astype('float32') / 255x_train = np.round(x_train, 0)x_test = np.round(x_test, 0)x_train = x_train.reshape((len(x_train), np.prod(x_train.shape[1:])))x_test = x_test.reshape((len(x_test), np.prod(x_test.shape[1:])))print(x_train.shape)
Note that we round up data. This is because this dataset is grayscale, we only consider strictly black pixels and strictly white pixels. This helps us form a binary cross-entropy as the loss function.
Let us create the network using Keras functional API.
inputs = tf.keras.layers.Input(784)encoded_1 = tf.keras.layers.Dense(128)(inputs)encoded = tf.keras.layers.Dense(64, activation='relu')(encoded_1)decoded_1 = tf.keras.layers.Dense(128)(encoded)decoded = tf.keras.layers.Dense(784, activation='sigmoid')(decoded_1)auto_encoder = tf.keras.Model(inputs, decoded)auto_encoder.compile(loss='binary_crossentropy',                      optimizer='adam',                      metrics=['accuracy'])auto_encoder.summary()tf.keras.utils.plot_model(auto_encoder, show_shapes=True, to_file='autoenc.png', dpi=200)
Note that I use binary_crossentropy since our data is in binary format (either zeros or ones). The complete set of layers is the autoencoder (encodes and decodes to give the same output). Let us have a look at our encoder. We can build it using the layers from inputs to encoded just as follows.
encoder = tf.keras.Model(inputs, encoded)encoder.summary()tf.keras.utils.plot_model(encoder, show_shapes=True, to_file='enc.png', dpi=200)
Although you can make this using the Sequential Keras API, I find the functional API more elegant (just the opinion, you might see why in the future articles).
Now that we have built our autoencoder, we can fit our training data.
auto_encoder.fit(x_train, x_train,                  epochs=10,                 batch_size=256,                 shuffle=True)
Once trained I obtained an accuracy of 0.9851 which is not so bad!. So we could expect a good performance in terms of dimension reduction.
Let us see the estimated lowed dimension and reconstructed images of the MNIST test dataset.
predicted_2dim = encoder.predict(x_test)predicted_original = auto_encoder.predict(x_test)
Note that for the decoded values, I straightaway feed the raw data to the complete autoencoder, which does the reconstruction straightaway after encoding. For the encodings, I use the encoder component.
The input data without being adjusted looks like below;
The encoded images look like this;
Reconstruction looks like this;
Note that the reconstructed image looks blurred. This happens since our last layer is activated by the sigmoid function. This function returns values in the range 0 and 1.
We use relu activation function because, in the middle layers, the chance of values getting diminished is higher when we use the sigmoid function.
When we decrease the hidden dimension the accuracy of reconstruction diminishes. The reconstruction looks like this. It only had the distinguishable features of characters, not the exact character.
We can see that in the above images 9 and 4 are almost similar. Much more information is lost.
My plotting function for characters (One might find useful)
shown = {}fig = plt.figure(figsize=(20, 10))fig.subplots_adjust(hspace=1, wspace=0.1)i = 0for data, y in zip(predicted_original, y_test):    if y not in shown and y==len(shown):        i += 1        ax = fig.add_subplot(1, 10, i)        ax.text(1, -1, str(y), fontsize=25, ha='center', c='g')        ax.imshow(np.array(data).reshape(28, 28), cmap='gray')shown[y] = True    if len(shown) == 10:        break
Feel free to change code and run. Hope this article was useful. I will introduce variational autoencoders in future articles(s)!