Residual Networks (ResNets). In earlier posts, we saw the... | by Muhammad Rizwan Khan | Towards Data Science

In earlier posts, we saw the implementation of LeNet-5, AlexNet, and VGG16 which are deep convolutional neural networks. Similarly, we can build our own deep neural network with more than 100 layers theoretically but in reality, they are hard to train. Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun introduced the concept of Residual Networks (ResNets) in their research Deep Residual Learning for Image Recognition. The ResNets allows a much deeper network to train efficiently.
What is the problem with very deep neural networks?During each iteration of training a neural network, all weights receive an update proportional to the partial derivative of the error function with respect to the current weight. If the gradient is very small then the weights will not be change effectively and it may completely stop the neural network from further training. The phenomenon is called vanishing gradients. More specifically we can say that the data is disappearing through the layers of the deep neural network due to very slow gradient descent.[1]
Microsoft research found that splitting a deep network into three layer chunks and passing the input into each chunk straight through to the next chunk, along with the residual output of the chunk minus the input to the chunk that is reintroduced, helped eliminate much of this disappearing signal problem. No extra parameters or changes to the learning algorithm were needed [1]. In other words, ResNets breaks down a very deep plain neural network into small chunks of network connected through skip or shortcut connections to form a bigger network.
Two main types of blocks are used in a ResNet, depending mainly on whether the input/output dimensions are same or different.
The identity block is the standard block used in ResNets and corresponds to the case where the input activation (say a[l]) has the same dimension as the output activation (say a[l+2]). Below is an example of identity block where the upper path is the “shortcut path” and the lower path is the “main path”.
When the input and output dimensions don’t match up, we add a convolutional layer in the shortcut path. The arrangement is called convolutional block.
You now have the necessary blocks to build a very deep ResNet. The following figure describes in detail the architecture of this neural network. “ID BLOCK” in the diagram stands for “Identity block,” and “ID BLOCK x3” means you should stack 3 identity blocks together.
The details of the above ResNet-50 model are:
Zero-padding: pads the input with a pad of (3,3)
Stage 1: The 2D Convolution has 64 filters of shape (7,7) and uses a stride of (2,2). Its name is “conv1”. BatchNorm is applied to the channels axis of the input. MaxPooling uses a (3,3) window and a (2,2) stride.
Stage 2: The convolutional block uses three sets of filters of size 64x64x256, f=3, s=1 and the block is “a”. The 2 identity blocks use three set of filters of size 64x64x256, f=3 and the blocks are “b” and “c”.
Stage 3: The convolutional block uses three sets of filters of size 128x128x512, f=3, s=2 and the block is “a”. The 3 identity blocks use three set of filters of size 128x128x512, f=3 and the blocks are “b”, “c” and “d”.
Stage 4: The convolutional block uses three sets of filters of size 256x256x1024, f=3, s=2 and the block is “a”. The 5 identity blocks use three set of filters of size 256x256x1024, f=3 and the blocks are “b”, “c”, “d”, “e” and “f”.
Stage 5: The convolutional block uses three sets of filters of size 512x512x2048, f=3, s=2 and the block is “a”. The 2 identity blocks use three set of filters of size 256x256x2048, f=3 and the blocks are “b” and “c”.
The 2D Average Pooling uses a window of shape (2,2) and its name is “avg_pool”.
The flatten doesn’t have any hyperparameters or name.
The Fully Connected (Dense) layer reduces its input to the number of classes using a softmax activation. Its name should be ‘fc‘ + str(classes).
from keras import layers from keras.layers import Input, Add, Dense, Activation, ZeroPadding2D, BatchNormalization, Flatten, Conv2D from keras.layers import AveragePooling2D, MaxPooling2D, GlobalMaxPooling2D from keras.models import Modelfrom keras.initializers import glorot_uniforminput_shape = (64, 64, 3) classes = 6# Define the input as a tensor with shape input_shape X_input = Input(input_shape)# Zero-Padding X = ZeroPadding2D((3, 3))(X_input)# Stage 1 X = Conv2D(64, (7, 7), strides = (2, 2), name = ‘conv1’,)(X) X = BatchNormalization(axis = 3, name = ‘bn_conv1’)(X) X = Activation(‘relu’)(X)X = MaxPooling2D((3, 3), strides=(2, 2))(X)# Stage 2 X = convolutional_block(X, f = 3, filters = [64, 64, 256], stage = 2, block=’a’, s = 1) X = identity_block(X, 3, [64, 64, 256], stage=2, block=’b’)X = identity_block(X, 3, [64, 64, 256], stage=2, block=’c’)# Stage 3 X = convolutional_block(X, f = 3, filters = [128, 128, 512], stage = 3, block=’a’, s = 2) X = identity_block(X, 3, [128, 128, 512], stage=3, block=’b’) X = identity_block(X, 3, [128, 128, 512], stage=3, block=’c’)X = identity_block(X, 3, [128, 128, 512], stage=3, block=’d’)# Stage 4 X = convolutional_block(X, f = 3, filters = [256, 256, 1024], stage = 4, block=’a’, s = 2) X = identity_block(X, 3, [256, 256, 1024], stage=4, block=’b’) X = identity_block(X, 3, [256, 256, 1024], stage=4, block=’c’) X = identity_block(X, 3, [256, 256, 1024], stage=4, block=’d’) X = identity_block(X, 3, [256, 256, 1024], stage=4, block=’e’)X = identity_block(X, 3, [256, 256, 1024], stage=4, block=’f’)# Stage 5 X = convolutional_block(X, f = 3, filters = [512, 512, 2048], stage = 5, block=’a’, s = 2) X = identity_block(X, 3, [512, 512, 2048], stage=5, block=’b’)X = identity_block(X, 3, [512, 512, 2048], stage=5, block=’c’)# AVGPOOL X = AveragePooling2D((2,2), name=’avg_pool’)(X)# output layer X = Flatten()(X)X = Dense(classes, activation=’softmax’, name=’fc’ + str(classes))(X)# Create model model = Model(inputs = X_input, outputs = X, name=’ResNet50′)#Compile the model model.compile(optimizer=’adam’, loss=’categorical_crossentropy’, metrics=[‘accuracy’])
Very deep neural networks (plain network) are not practical to implement as they are hard to train due to vanishing gradients.
The skip-connections help to address the Vanishing Gradient problem. They also make it easy for a ResNet block to learn an identity function.
There are two main types of ResNets blocks: The identity block and the convolutional block.
Very deep Residual Networks are built by stacking these blocks together.
https://en.wikipedia.org/wiki/Vanishing_gradient_problem
Convolutional Neural Network by Andrew Ng (Coursera)
Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun — Deep Residual Learning for Image Recognition (2015)
Francois Chollet’s GitHub repository: https://github.com/fchollet/deep-learning-models/blob/master/resnet50.py