CNN Architectures from Scratch
By: Sean Benhur J
Tags: Artificial Intelligence, Convolutional Network, Machine Learning, Data Science, Deep Learning

CNN Architectures from Scratch

Photo by James Padolsey on Unsplash

Convolutional Neural networks are a class of Deep Neural Networks, which achieve State of the Art results not only in Computer Vision tasks but also in other fields such as Speech recognition, Natural Language Processing, etc. These CNNs have evolved in a long way by not only stacking layers but also creating custom layers. In the current world, we use these models to transfer learning and achieve better results.

In this article, I will write how to implement state of the art CNN architectures from scratch in Pytorch. I assume that you know the theory about these CNNs, so that I won’t be covering the theory part. At the end of this post, I will also include some additional resources to learn about the theory.

Lenet:

Lenet 5 is considered as the first architecture for Convolutional Neural Networks, which are used to identify handwritten digits in the zip codes in the US. It was introduced in the paper, “Gradient-Based Learning Applied To Document Recognition.”

Lenet Architecture, Image Source

The LeNet-5 architecture consists of two sets of convolutional, activation, and pooling layers, followed by two fully-connected layers, tanh activation, and finally a softmax classifier which classifies the MNIST digits[1]

Now let’s implement the Lenet in Pytorch.

Lenet Pytorch Implementation

In the above code, we create a Neural network with tanh activation function, two Convolutional layers followed by the AveragePooling layer, a Convolutional layer followed by two linear layers with a softmax classifier [2].

Alexnet:

AlexNet was a deep neural network that was developed by Alex Krizhevsky and others in 2012. It was designed to classify images for the ImageNet [3] ILSVRC-2012 competition, where it achieved the state of the art results. It was introduced in the paper ImageNet Classification with Deep Convolutional Neural Networks

Alexnet starts with an input layer of 227 x 227 x 3 images , the next convolution layer consists of 96 (11 x 11) filters with a stride of 4. which reduces its dimension by 55 x 55. Followed by a MaxPool layer with 3 x 3 filter along with stride 2. This process goes on and finally reaches the Fully Connected layer with 9216 parameters and the next two Fully Connected layers with 4096 nodes each. In the end, it uses the Softmax function with 1000 output classes.. Alexnet is the first architecture to use ReLU non-linearity , Dropout for regularization and Multi-GPU training[3]

Now let’s implement the Alexnet in Pytorch.

In the above code, we create a Neural Net with the first convolutional layer of kernel size = 11, stride = 4, and padding = 2, followed by a LocalResponseNormalization layer and Pooling layer, from the second layer LocalResponseNormalization layer is avoided, and up to five convolutional layers are created, followed by three Fully connected Layers which outputs 1000 classes at the end.

VGG:

VGG can be called a deeper form of Alexnet. This network stacks more layers than Alexnet and uses the same ReLU activation function but has a lesser number of parameters compared to Alexnet.

VGG net architecture Image Source

VGG-16 is a simpler architecture model since it's not using much hyperparameters. It always uses 3 x 3 filters with a stride of 1 in the convolution layers and uses the SAME(1,1) padding in pooling layers with a stride of 2.[3]

Now let’s implement VGG in Pytorch.

In the above code, first, we create a dictionary of layer values in which ‘M’ stands for MaxPool layers, and the integer numbers are the number of input channels and output channels, next. We create a function create_conv_layers, which creates a number of convolutional layers with the help of the dictionary, and the __init__ method and __forward__ method are as same as any other CNN.

InceptionNet(GoogleNet):

The Inception network is also considered as Googlenet, which is considered an important milestone in the history of CNNs. Before the inception net, all the networks were stacking the layers, aiming to get better performance but failed. The Inception network was the winner of the Imagenet 2014 challenge and was introduced in the paper “Going deeper with Convolutions.” Its main contribution was the development of an Inception Module that dramatically reduced the number of parameters in the network (4M, compared to AlexNet with 60M). Additionally, this paper uses Average Pooling instead of Fully Connected layers at the top of the ConvNet, eliminating a large number of parameters that do not seem to matter much [4].

Inception network architecture Source

The architecture also contains two auxiliary classifier layers(yellow block) connected to the output of Inception modules.

Now let’s implement the convolutional block of the Inception module, which will be the building blocks of the whole Inception Architecture.

In the above code, we create a convolutional layer with ReLU activation function and BatchNorm for regularization. The convolutional layer gets a number of input and output channels dynamically.

The implementation of Auxillary classifiers as follows,

In the above code, we create an average pooling layer of 5*5 filters and convolutional layers of 1*1 filters followed by Linear layers.

Now let's implement the Inception module.

The Inception Module described in the paper

The above code is based on the flowchart, in which we create four branches of convolutions and concatenate them, which in turn forms the Inception Module.

Now we can implement the full inception architecture.

In the above code, first, we create two convolutional layers of kernel sizes 3*3, followed by Inception Blocks. The above code can be better understood with the help of this table.

The main feature of these Inception blocks are, these are not only deeper but also wider

Resnet:

Image from the paper

Resnet is considered as a game-changing architecture because it is considered as a real deeper architecture which has 152 layers. It was introduced in the paper “Deep Residual Learning for Image Recognition” it won the Imagenet 2015 competition, ever since most of the CNNsare variants of these Resnets. It became so popular since it avoids the degradation problem, that is when creating deeper nets mostly loss of the network increased after a certain point both in train and test sets, to avoid this major problem, the authors proposed to use a reference to the previous layer [5] to compute the output at a given layer. In ResNet, the output from the previous layer, which is called residual, is added to the output of the current layer.

Identity connection Image source

The implementation of Resnet as follows; first, we create a block of layers, which helps us to create the main network.

The above code creates a block of convolutional layers with BatchNormalization with ReLU activation and skip-connection, which is known as identity_downsample, now. Let’s implement the main Resnet layer with skip connection, which will be used in the main network

It can be defined by stacking n blocks one after the other, we should note that the first convolutional block has a stride of 2 since "We are performing identity downsampling directly by convolutional layers which have a stride of 2 [5].

With the help of the above functions, we can create our full Residual network from scratch, which follows the original implementation from the paper “Deep Residual Learning for Image Recognition.”

The above code creates the full ResNet model. The main advantage of this architecture is we can train a deeper network with this architecture