Inception Network and Its Derivatives
By: Ritacheta Das
Tags: Machine Learning, Deep Learning, Data Science, Artificial Intelligence

In a traditional image classification model, each layer extracts information from the previous layers in order to get useful information. However, each layer type extracts different kinds of information. The output of a 5x5 convolutional kernel tells us something different from the output of a 3x3 convolutional kernel, which tells us something different from the output of a max-pooling kernel, and so on and so on. But, how can you be sure whether the transformation is giving useful information or not?

We mostly try deeper network models. But, what if we try some wider network.

Let’s dive in to discover the answers.

The answer is Inception Network. The inception network has played an important role in the world of ML. The Inception network is engineered too much to compete with speed and accuracy.

In traditional neural networks due to working just with the previous layers causes loss of useful data. Inception models use the same input data to learn by computing multiple different conversions parallelly, concatenating them in a single output. Simply saying, Inception does 1✕1 convolutional transformation, 3✕3, 5✕5, and more others and some pooling layers. Then stack all the outputs of the transformation and let the model choose how to use that information.

Before diving into the types of inception, let’s gather some knowledge about 1✕1 convolution.

1✕1 convolution is simply used to reduce the computation to an extent. Let’s see an example for better understanding.

Assume you need to compute a 3✕3 convolution operation with and without using 1✕1 convolution.

Without 1✕1 convolution total number of operations involved is = (3✕3✕300)✕(14✕14✕30) = 15876000

Using 1✕1:

Using 1✕1 convolutional total number of operation involved is = (14✕14✕10)✕(1✕1✕300) + (14✕14✕30)✕(3✕3✕10) = 1117200

Due to using 1✕1 convolution 14M operations are reduced. Thus, 1×1 convolution can help to reduce model size which can also somehow help to reduce the overfitting problem.

There is another term I have used that is Inception module. So let’s know what the inception module really is.

Inception Module:

Inception Modules are used in Convolutional Neural Networks to allow for more efficient computation and deeper Networks through dimensionality reduction with stacked 1×1 convolutions. The modules were designed to solve the problem of computational expense, as well as overfitting, among other issues. In the Inception module 1×1, 3×3, 5×5 convolution and 3×3 max-pooling performed in a parallel way at the input, and the output of these are stacked together to generate the final output. The idea behind that convolution filters of different sizes will handle objects at multiple scales better.

Inception Network consists of multiple versions.

Inception v1

Using this inception module with dimensionality reduction a neural network is architected. The most simple neural network made up of this way is known as Inceptionv1 or Google Net. The architecture is shown below.

GoogLeNet has 9 such inception modules stacked linearly. It is 22 layers deep and 27, including the pooling layers. It uses global average pooling at the end of the last inception module. Due to its lesser loss of data its classification ability is truly good and can prevent the vanishing gradient problem. GoogleNet or inceptionv1 was the winner of ILSVRC (ImageNet Large Scale Visual Recognition Competition) 2014, an image classification competition.

Thus, Inception Net emerged victorious over the previous versions of CNN models. It achieves an accuracy of top-5 on ImageNet, it reduces the computational cost to a great extent without compromising the speed and accuracy.

Problems in Inception v1- InceptionV1 use 5✕5 convolution that causes loss of information. Furthermore, there is also a complexity decrease when we use bigger convolutions like 5×5 as compared to 3×3. We can go further in terms of factorization i.e. that we can divide a 3×3 convolution into an asymmetric convolution of 1×3 then followed by 3×1 convolution. This is equivalent to sliding a two-layer network with the same receptive field as in a 3×3 convolution but 33% cheaper than 3×3. This factorization does not work well for early layers when input dimensions are big but only when the input size is m×m where m is between 12 and 20.

Inception v2: Inception v2, 3⨯3 convolutions is used in the space of 5⨯5 and boosts the performance. This also decreases computational time and thus increases computational speed because a 5×5 convolution is 2.78 times more expensive than 3×3 convolution. So, Using two 3×3 layers instead of 5×5 increases the performance of architecture.

figure1

The architecture also converts n×n factorization into 1xn and nx1 factorization as shown below.

figure2

To deal with the problem of the representational bottleneck, the feature banks of the module were expanded instead of making it deeper. This would prevent the loss of information that causes when we make it deeper.

figure3

Using these 3 above ideas the inception v1 architecture is updated in inception v2. Below is the layer-by-layer details of Inception v2:

Notice in the above architecture figures 5, 6, 7 refers to figures 1, 2, 3 in this article.

Inception v3: Inception v3 is almost similar to Inception v2 except for some updates. Those updates are listed below:

Use of RMSprop optimizer.

Batch Normalization in the fully connected layer of the Auxiliary classifier.

Use of 7×7 factorized Convolution.

Label Smoothing Regularization(a method to regularize the classifier by estimating the effect of label-dropout during training). It prevents the classifier to predict too confidently a class. The addition of label smoothing gives 0.2% improvement from the error rate.

Inception V4:

The main motto behind creating inception v4 is to reduce the complexity of inception v3 and make the module more uniform.

This is a pure Inception variant without any residual connections. It can be trained without partitioning the replicas, with memory optimization to backpropagation.

Inception-ResNet v1 and v2: Inspired by the success of ResNet, a combination of inception and the residual module was proposed. There are two models in this combination: Inception ResNet v1 and v2.

Using the inception module in the above architecture the computation cost decreases. Each Inception block is followed by a 1×1 convolution without activation called filter expansion. This is done to scale up the dimensionality of the filter bank to match the depth of input to the next layer. In Inception ResNets models, the batch normalization is not used after summations. This is done to reduce the model size to make it trainable on a single GPU. Both the Inception architectures have the same architectures for Reduction Blocks but have different stem structures. They also have differences in their hyperparameters for training. It is found that Inception-ResNet V1 has similar computational cost as of Inception V3 and Inception-ResNet V2 have similar computational cost as of Inception V4. The total structure of Inception ResNetv2 is shown below:

Results: With Single-Crop Single-Model the top-5 and top-1 error rate on the ILSVRC 2012 validation sets are below:

With 10/12-Crop Single-Model, again, Inception-v4 and Inception-ResNet-v2 again have the best performance with similar results.

The top-5 and top-1 error rate of 144-crop single-model evaluation of different architectures on the ILSVRC 2012 validation sets are below:

Resources: