Using Skip Connections To Enhance Denoising Autoencoder Algorithms | by Metika Sikka | Towards Data Science

The official Keras blog, calls autoencoders an example of ‘self-supervised’ algorithms as their targets are generated from the input data. Hence, they are used for tasks of image reconstruction.
The main parts of an autoencoder are: Encoder, Bottleneck and Decoder. The Encoder is extracts image features at each step and in the process compresses the input data. The bottleneck constrains the input to its lowest dimensions known as compressed representations of the input data. The Decoder comes after this bottleneck and is used to reconstruct the input data. A loss measure is used to compare how close the generated or reconstructed date is to the input data.
Data denoising images is a common application of autoencoders. Noise in images can be understood as a random variation in color or brightness of images, degrading their quality. Removing this noise is often a pre-processing step in various use-cases of image data. Convolutional autoencoders can be used for this purpose. The encoder learns to extract the features separating them from the noise in the image. Thereby compressing the image. The final compressed representations from the bottleneck are passed over to the decoder. The decoder finally decompresses the image minimizing the noise.
We know that deep neural networks suffer from the degradation problem. Since Autoencoders have multiple convolutional and deconvolutional layers, they also suffer in performance when reconstructing images due to this information loss. Residual networks comprising of skip connections are a known solution to this problem. Hence, to improve the performance of the autoencoders such ‘skip connections’ can be added from the encoder to the decoder, i.e. across the bottleneck. These additional connections can directly send the feature maps from the an earlier layer of the encoder to a later layer of the decoder. This helps the decoder form more clearly defined decompressions of the input image.
To see the advantage of these residual networks, let’s see the activation outputs at various stages of a convolutional autoencoder model. First we trained an autoencoder model without skip-connections across the bottleneck. Then we added them and train the model again on the same sample of images.
A sample of 10K RGB images were taken from this data set of Flickr images on Kaggle. All the analysis was done using Google Colab’s GPU.
We have used the Functional API of Keras to form the autoencoder model. The input images were resized to a smaller size (128,128,3) because of computational limitations. The image pixels were also standardized by a factor of 1/255. Further, the images were corrupted on purpose by applying a Gaussian noise matrix. These corrupted images formed the input of the autoencoder whereas the original images were used as targets while training the model. When building convolutional networks it is important to remember that, as we go deeper, the number of channels or filters increases whereas the size (height and width) of the input decreases.
#Input input_img = Input(shape=(128, 128, 3))#Encoder y = Conv2D(32, (3, 3), padding='same',strides =(2,2))(input_img)y = LeakyReLU()(y)y = Conv2D(64, (3, 3), padding='same',strides =(2,2))(y)y = LeakyReLU()(y)y1 = Conv2D(128, (3, 3), padding='same',strides =(2,2))(y) # skip-1y = LeakyReLU()(y1)y = Conv2D(256, (3, 3), padding='same',strides =(2,2))(y)y = LeakyReLU()(y)y2 = Conv2D(256, (3, 3), padding='same',strides =(2,2))(y)# skip-2y = LeakyReLU()(y2)y = Conv2D(512, (3, 3), padding='same',strides =(2,2))(y)y = LeakyReLU()(y)y = Conv2D(1024, (3, 3), padding='same',strides =(2,2))(y)y = LeakyReLU()(y)#Flattening for the bottleneckvol = y.shapex = Flatten()(y)latent = Dense(128, activation='relu')(x) 
Below is code for the decoder. Transposed convolution or deconvolutional layers were used to build this decoder. Deconvolutional layers work roughly like a combination of convolutional and upsampling layers. Initially, we trained the model without the first and second skip connections. Then these connections were added from an earlier layer of the encoder to the later layer of the decoder using Add() from the layers API in Keras. The lrelu_bn() helper function was used to apply activations to these additions and pass the result through a batch normalization layer.
# Helper function to apply activation and batch normalization to the # output added with output of residual connection from the encoderdef lrelu_bn(inputs):   lrelu = LeakyReLU()(inputs)   bn = BatchNormalization()(lrelu)   return bn#Decodery = Dense(np.prod(vol[1:]), activation='relu')(latent)y = Reshape((vol[1], vol[2], vol[3]))(y)y = Conv2DTranspose(1024, (3,3), padding='same')(y)y = LeakyReLU()(y)y = Conv2DTranspose(512, (3,3), padding='same',strides=(2,2))(y)y = LeakyReLU()(y)y = Conv2DTranspose(256, (3,3), padding='same',strides=(2,2))(y)y= Add()([y2, y]) # second skip connection added herey = lrelu_bn(y)y = Conv2DTranspose(256, (3,3), padding='same',strides=(2,2))(y)y = LeakyReLU()(y)y = Conv2DTranspose(128, (3,3), padding='same',strides=(2,2))(y)y= Add()([y1, y]) # first skip connection added herey = lrelu_bn(y)y = Conv2DTranspose(64, (3,3), padding='same',strides=(2,2))(y)y = LeakyReLU()(y)y = Conv2DTranspose(32, (3,3), padding='same',strides=(2,2))(y)y = LeakyReLU()(y)y = Conv2DTranspose(3, (3,3), activation='sigmoid', padding='same',strides=(2,2))(y)
Since the input images were standardized, their resulting pixel values were between 0 and 1. To get a comparable reconstructed image, “sigmoid” activation was used in the final layer. Out of the training set, 1k images were used as the validation set. Binary cross entropy was used as the loss function. Finally, both models were trained for 200 epochs with a mini-batch size of 32. The Adam optimizer with a learning rate of 0.001 gave the least training and validation loss at convergence.
Let’s look at the activation outputs at various layers of the model for a test image. This will help us clearly see the encoder-decoder in action!
The activation outputs were created using the keract package.
We take a test image of the Columbia University campus. The test image was resized for the model and random noise was added. The ‘noisy’ image was used as the input for the model.
Since the encoder was same for both the models, the activation outputs were also the same for this part. On examining the activation outputs of the first three convolutional layers, it can be seen that most of the image information is retained. This could also imply that the noise in the image is also retained at this stage of the model. However, as we go deeper in the model (see activation outputs of layers after the third convolutional layer), the information retained is quite abstract. The model starts to extract higher level features such as borders, corners & angles.
It is interesting to note that the activation outputs of the decoder are in the exact opposite sequence as the encoder above.
After the third de-convolutional layer, we still do not see any edges of the image being formed again.
There is no clearly defined decompression of the image in the final layer of the decoder as seen below!
As seen in the activation outputs of the encoder, its earlier layers retain the noise but the later layers extract higher representations of the image. Hence, skip connections are made from the third and fifth convolutional layers of the encoder to third and fifth deconvolutional layers of the decoder. The activation outputs of the first, second and third de-convolutional layers for this decoder are the same as the previous version of the decoder. However, after the third layer we see a difference in the activation outputs from those of the previous decoder. Below we can clearly see the image being formed again in the activation output of the sixth deconvolutional layer.
The final layer of this decoder has given a clearly defined decompressed version of the test input image.
Denoised image output from the first model:
Denoised image output of the second model with skip connections across the bottleneck:
We can clearly see the better performance the autoencoder model with skip connections across the bottleneck!
This approach can be used for any image reconstruction application of autoencoders apart from denoising images. While the skip connections improve the performance of the autoencoder, the positions and number of these connections can be experimented with. The model can also be trained with different levels of noise factor for better generalization of results. The full code can be accessed here!
J. Dong , XJ. Mao , C. Shen , YB. Yang, Learning Deep Representations Using Convolutional Auto-encoders with Symmetric Skip Connections (2017)