CNN Architectures, a Deep-dive. Implementing every popular CNN... | by Sai Kumar Basaveswara | Towards Data Science

In Deep Learning, a Convolutional Neural Network(CNN) is a class of deep neural networks, most commonly applied to analyzing visual imagery. Convolutional Neural Networks are state of the art models for Image Classification, Segmentation, Object Detection and many other image processing tasks. In order to get started into the field of Image Processing or to improve the prediction accuracy of the custom CNN models, the knowledge of some of the famous CNN architectures will keep us reach the pace in this competitive and ravenous world. Ergo, in this post I will be covering a few famous CNN architectures. I mainly stress on how to implement them in Keras in general i.e., making custom models with the idealogy and structure of those architectures.
In this post, I will be covering the following architectures:
VGG Net
ResNet
Dense Net
Inception Net
Xception Net
VGG Net is a plain and straight forward CNN architecture among all other. Thought it looks simple, it do outperform many complex architectures. It is the 1st runner-up in ImageNet Challenge in 2014. As shown above, there are totally 6 VGGNet Architectures. Among them, VGG-16 and VGG-19 are popular.
The idea of VGG architectures is quite simple. we have to stack the convolutional layers with increasing filter sizes. i.e., if layer-1 has 16 filters, then layer-2 must have 16 or more filters.
Another noteworthy point is that in every VGG architecture, all filters are of size 3*3. The idea here is that, two 3*3 filters almost cover the are of what a 5*5 filter would cover and also two 3*3 filters are cheaper than one 5*5 filter(cheaper in the sense of total no. of multiplications to be performed).
Lets create a custom VGG Net with 6 convolutional layers in Keras.
The above code will create a simple VGG Network with six convolutional layers. After every Convolutional layer, I added a Dropout layer to reduce overfitting and MaxPooling layer after every pair of Convolutional layers for Dimensionality Reduction.
The problem with VGG is that this naive architecture is not good for deeper networks, as the deeper the network goes, it is more prone to Vanishing Gradients Problem. More training and also more parameters have to be tuned in deeper VGG architectures.
However, VGG Nets are handy for transfer learning and small classification tasks.
Residual Networks are the first deeper networks to be won on the ImageNet Challenge. The ResNet used at ImageNet in 2015 has 152 layers. Until then, the idea of training such deeper networks is a dream. ResNet however achieved an error rate of 3.57% (top-5 error rate).
The success recipe of ResNet for training such a deep(152 layers) network is that, it has residual connections. In VGG, every layer is connected to its previous layer, from which it is getting its inputs. This makes sure that upon propagation from layer to layer, more and more useful features are carried and less important features are dropped out. This is not the best way since the latter layers cannot see what the former layers have seen. ResNet addresses this problem by connecting not just the previous layer to the current layer but also a layer behind the previous layer. By incorporating this, now each layer can see more than just its previous layer’s observations.
There are many variants of ResNets. The core idea is, let us consider x as an output of some Conv2D layer. Add few Conv2D layers to x and then add the output to x and send this as input to the next layer.
Training of such a deep Residual Networks is possible by using BatchNormalisation layers after every Convolutional Layer. Batch Normalisation layers will boost the values of weights and hence higher learning rates can be used while training, which will help train faster and also can minimize Vanishing Gradient problem.
Lets convert the notion of ResNet into a snippet of code, which can be arbitrary to achieve the architecture we desire.
The above function takes arguments as follows:
x : input to the res_layer.filters : number of filters in each convolutional layer.pooling : whether to add a pooling layer (default is False).dropout : whether to add a dropout layer (default is No).
And the process is, it connects the input layer to a Conv2D layer having filters specified in our function calling and then attaches a BatchNormalization layer, upon which a ReLU Activation layer is added, and then another Conv2D layer is stacked. Now, this stacked output is added with the initial input, but the initial input is transformed by passing it through a Conv2D layer of given filters. This step is done to match the output sizes of both the layers that are going to be added. Then, if we prefer having a MaxPooling2D layer, we add that and also if any Dropout value is given, then a Dropout layer is also added and then finally one more BatchNormalization layer and Activation layer are added and then this final layer is returned by our res_layer function.
Now that we have created our res_layer, lets create a custom resnet.
The above code created a simple ResNet model. The res_layer function is used to simplify the process of stacking multiple layers and adding them multiple times, which makes our code readable and manageable.
The advantage of ResNet is that we can train deeper networks with this architecture.
In ResNet, we added the stacked layer along with its input layer. In DenseNet, for a given layer, all other layers preceding to it are concatenated and given as input to the current layer. With such an arrangement, we can use smaller filter counts and also, this will minimize the vanishing gradient problem as all layers are directly connected to the output, gradients can be calculated directly from the output for each layer.
Similar to res_layer function, lets develop a function for dense_layer.
Inputsx : input layer.layer_configs: a list of dictionaries, where each dictionary is of the below format.
Inside the layer_configs, the dictionaries have the following keys:"layer_type" : which type of layer we are going to create. Currently only Conv2D layer is supported by the above dense_layer function."filters" : determining number of filters in that layer. An integer is given as value."kernel_size" : size of the kernel. A tuple of kernel size is given as value. like (3, 3)."strides" : step size of stride. An integer is given as value."padding" : type of padding to be applied to the layer."activation" : type of activation to be applied to the layer.
Now that we have defined our function, lets create some custom layer_configs.
Now, lets create a custom DenseNet.
DenseNet have much more intermediate connections when compared to ResNet. Also, we can use smaller filter counts in Dense layers and is good for smaller models.
Inception means going deeper. In ResNet, we created deeper networks. The idea of Inception Net is to make the network wider. This can be done by parallel connection of multiple layers having different filters, and then finally concatenating all of those parallel paths to pass to next layers.
We can do that by writing the inception_layer function, which can create an inception_layer of arbitrary configurations.
Inputsx : input layer.layer_configs : a list of lists, where each list have dictionaries.
Let us see a demo layer_configs list to get the idea.
The keys in the dictionaries are:"layer_type" : which type of layer we are going to create. Currently only Conv2D layer is supported by the above dense_layer function."filters" : determining number of filters in that layer. An integer is given as value."kernel_size" : size of the kernel. A tuple of kernel size is given as value. like (3, 3)."strides" : step size of stride. An integer is given as value."padding" : type of padding to be applied to the layer."activation" : type of activation to be applied to the layer.
Now that we have created a inception_layer function, lets create a custom Inception Net. The layer_configs for different layers in this example can be found in this gist.
Finally, lets create a custom Inception Net.
There are many variants of Inception Nets. The differences among them are:
Instead of using a 5*5 filter, use two 3*3 filters as they are computationally efficient (as discussed in the VGGNet).
Using a 1*1 Conv2D layer with smaller filter count before performing any Conv2D layer with larger filter sizes as 1*1 filter with less filter count will reduce the depth of the input and hence is computationally efficient.
Instead of performing a 3*3 filter, perform 1*3 filter followed by a 3*1 filter. This will drastically improve the computational efficiency.
Using our inception_layer function, we can customize all above types of InceptionNet architectures by writing our layer_configs accordingly.
InceptionNets are preferable as they are not just deeper, but also wider and we can stack many such layers and still the output params to be trained are less when compared to all other architectures.
Xception Net is an improvisation of InceptionNet in terms of computational efficiency. Xception means Extreme Inception. The Xception architecture presented in the image above is more like a ResNet rather than an InceptionNet. Xception Net outperforms Inception Net v3.
The difference between Inception Net and Xception Net is that, in Inception Net normal convolutional operations are performed whereas in Xception Net, Depthwise Separable Convolutional operations are performed. Depthwise Separable Convolutions are different from normal convolutions in a way that, in normal Conv2D layer, for an input of (32, 32, 3) image we can use any number of filters in the Conv layer. Each of those filters will be operated over all three channels and the output is the sum of all corresponding values. But in Depthwise separable convolutions, each channel have only one kernel to do convolution. Hence, by performing Depthwise Separable Convolutions, we can reduce the computational complexity as every kernel is of two dimensional only and is convoluting only over one channel. In Keras, we can implement this by using DepthwiseConv2D layer.
Lets create a simple Xception Net architecture.
The above is a simpler implementation of Xception Net architecture. We can use XceptionNet in low power devices as there are less compuations in the Conv layer and also is accuracy is pretty similar when compared to a normal convolution layer.
The main idea behind this article is to get familiar with state-of-the-art CNN architectures by actually creating them. This will build and improve our intuition and understanding about CNN’s and how to and when to use them.There are many other popular CNN architectures. They appear more or less similar to the above architectures with added functionalities.
For example, let us take MobileNet. It performs Depthwise Separable Convolutions rather than normal convolutions. This makes it more suitable for using it in low power devices and for models with faster responses.
To wrap up everything, I prefer using custom architectures like these in smaller classification tasks with decent dataset sizes and moves to Transfer Learning in case of smaller datasets. When speed of response is concerned, we move for smaller models. Also, by using Ensemble methods, we can get pretty good accuracy with custom architectures.
The articles and papers that helped me do this are:
Inception Net https://arxiv.org/pdf/1409.4842v1.pdf
Dense Net https://arxiv.org/pdf/1608.06993v3.pdf, https://towardsdatascience.com/densenet-2810936aeebb
Xception Net http://zpascal.net/cvpr2017/Chollet_Xception_Deep_Learning_CVPR_2017_paper.pdf