Is the Transposed Convolution layer and Convolution layer the same thing? Experimenting with concepts using PyTorch. | by Marios Aspris | Towards Data Science

This article has first appeared here.
So I was looking at a paper that was asking the same thing, but the original title of the paper is: Is the deconvolution layer the same as a convolutional layer? . You can see the article details at the link. So I got curious about this and I went through it. The original paper talks about improving the resolution of an image by upsampling the image with Transposed Convolution from the low resolution space, and also examines and compares there method on sub-pixel convolution.
In this post I will not test these concepts about improving the resolution of an image, rather I am going to test and compare a Neural Network for a classification task using the regular convolution operations and compare the same Network by replacing the convolutions with transposed convolutions. I will follow the same architecture on the Densely Connected Convolutional Network that I have created and describe on my previous blog post.
The transpose convolution operation is very well known by now and has been used in many models where upsampling is needed. It is very similar to the convolution operation, only that the convolution matrix is transposed. Therefore the result is that the output grows instead of reducing(depending on the padding, stride and kernel size used). A very well mathematical explanation of the convolution arithmetic can be found at this link. But first, let’s run some tests on how the output would look like of convolution transpose on random tensors.
First, I import PyTorch and wrap into Variable, a random tensor sampled from the normal distribution of batch size 1, channel dimension 1 and height, width 5, 5 respectively as shown below. Then I define the transpose convolution operation to take the right inputs, with kernel size 3x3, stride 1 and padding 0. Translating these into PyTorch code:
import torch import torch.nn as nn from torch.autograd import Variable random_tensor = Variable(torch.randn(1, 1, 5, 5)) upsample_tnsr = nn.ConvTranspose2d(in_channels = 1, out_channels = 1, kernel_size = 3, stride = 1, padding = 0)
Now before moving on, I would like to specify some notations, to easily follow the definitions.
Some settings:
$latex n_W = $ the input dimension of the width, and $latex n_H = $ the input dimension of the height. Since the tensor will be square we will just have the parameter n representing both.
p = is the padding applied to the tensor
f = is the size of the square kernel
s = is the stride that will be applied with the convolution operation.
c = is the channel dimension of the tensor
If we apply a regular convolution on a tensor the outcome can be described as follows:
This will be the output of the Width and Height of the tensor, and it will be the same since we can assume that the input will have the same height and width dimension.
As an example, suppose I consider the above specified tensor, applying regular convolution operation on it with padding p = 1, kernel size f =3 and stride s = 1, our output will be the same as the input n.
Now, if we want to run the transpose convolution on this, then the outcome we will expect, is described by the following equation:
Suppose on the same tensor, we run a transpose convolution with stride s = 2, kernel f = 3, and padding = 1. Then the output we will expect will be output = 9.
Now if we use stride = 1, kernel size f = 3 and padding = 1, then the result of the output will be the same as the input.
Below you can see the results:
Now I make use of the fact that the output of a transpose convolution, with the right settings stays the same as the input. This will be the hidden layers of the network, and the input picture to the network will only get reduced with the transition layers due to the average pooling operation.
So a Dense Block for the network will look like this:
class Dense_Block_Transposed(nn.Module):   def __init__(self, in_channels):    super(Dense_Block_Transposed, self).__init__()     self.relu = nn.LeakyReLU(inplace = True)    self.bn = nn.BatchNorm2d(num_features = in_channels)     self.conv1 = nn.ConvTranspose2d(in_channels = in_channels, out_channels = 32, kernel_size = 3, stride = 1, padding = 1)    self.conv2 = nn.ConvTranspose2d(in_channels = 32, out_channels = 32, kernel_size = 3, stride = 1, padding = 1)    self.conv3 = nn.ConvTranspose2d(in_channels = 64, out_channels = 32, kernel_size = 3, stride = 1, padding = 1)    self.conv4 = nn.ConvTranspose2d(in_channels = 96, out_channels = 32, kernel_size = 3, stride = 1, padding = 1)    self.conv5 = nn.ConvTranspose2d(in_channels = 128, out_channels = 32, kernel_size = 3, stride = 1, padding = 1)  def forward(self, x):    bn = self.bn(x)       conv1 = self.relu(self.conv1(bn))    conv2 = self.relu(self.conv2(conv1))       c2_dense = self.relu(torch.cat([conv1, conv2], 1))    conv3 = self.relu(self.conv3(c2_dense))       c3_dense = self.relu(torch.cat([conv1, conv2, conv3], 1))    conv4 = self.relu(self.conv4(c3_dense))       c4_dense = self.relu(torch.cat([conv1, conv2, conv3, conv4], 1))     conv5 = self.relu(self.conv5(c4_dense))       c5_dense = self.relu(torch.cat([conv1, conv2, conv3, conv4, conv5], 1))      return c5_dense 
Here, the height and width dimension of the image does not change, only the channel dimension increases.
Next building block of the network, is the Transition Layer. I kept the same structure of the Transition layer in the Dense Net on my previous post, with a minor change of defining a transpose convolution.
So the code for the Transition Layer would look like this:
class Transition_Layer_Transposed(nn.Module):  def __init__(self, in_channels, out_channels):     super(Transition_Layer_Transposed, self).__init__()        self.relu = nn.LeakyReLU(inplace = True)    self.bn = nn.BatchNorm2d(num_features = in_channels)       self.conv = nn.ConvTranspose2d(in_channels = in_channels, out_channels = out_channels, kernel_size = 1, bias = False)     self.avg_pool = nn.AvgPool2d(kernel_size = 2, stride = 2, padding = 0)     def forward(self, x):    bn = self.bn(self.relu(self.conv(x)))    out = self.avg_pool(bn)       return out
Now to I train this model on CIFAR10 with the exact architecture as the one I described on my previous post. So I train both networks with Stochastic Gradient Descent and the exact same hyperparameters. I run the optimization for 60 epochs and I plot the loss on the training data.
Now let us see how these networks perform on the test data, and their accuracy in each class individually:
                   Transposed Convolutions || Regular Convolutions Accuracy of plane :          84 %                    85 % Accuracy of car   :          89 %                    88 % Accuracy of bird  :          67 %                    72 % Accuracy of cat   :          60 %                    62 % Accuracy of deer  :          76 %                    71 % Accuracy of dog   :          74 %                    72 % Accuracy of frog  :          83 %                    83 % Accuracy of horse :          83 %                    79 % Accuracy of ship  :          89 %                    88 % Accuracy of truck :          88 %                    86 % 
From the table above it seems that the two networks have comparable accuracy on the test set, meaning that both were able to learn meaningful results.
From the experiment above transpose convolutions seem to learn meaningful results and have comparable accuracy with regular convolutions, even for classification tasks.
But transpose convolution is used more for upsampling purposes, in Generative Adversarial Networks for example. But as this article suggests the are better uses of upsampling layers, like running bi-linear interpolation to upsample and then a convolution. One example you can observe this is described by the paper Feature Pyramidal Networks for Object Detection.
To see my full code and reproduce my results, see my full code on this link.
Is the deconvolution layer the same as a convolutional layer?Deconvolution and Checkerboard ArtifactsFeature Pyramid Networks for Object Detection
Is the deconvolution layer the same as a convolutional layer?
Deconvolution and Checkerboard Artifacts
Feature Pyramid Networks for Object Detection