Components of convolutional neural networks | by John Olafenwa | Towards Data Science

Recent state-of-the-art architectures have employed a number of additional components to complement the convolution operation. In this post, I would be explaining some of the most important components that have improved both the speed and accuracy of modern convolutional neural networks. I would begin by explaining the theory of each of the components and finalize with a practical implementation in keras.
The first secret sauce that has made CNNs very effective is pooling. Pooling is a vector to scalar transformation that operates on each local region of an image, just like convolutions do, however, unlike convolutions, they do not have filters and do not compute dot products with the local region, instead they compute the average of the pixels in the region (Average Pooling) or simply picks the pixel with the highest intensity and discards the rest (Max Pooling).
The above is a 2 x 2 pooling, it will effectively reduce the size of feature maps by a factor of 2.
The very idea of pooling can seem counter-productive as it leads to loss of information, however it has proven to be very effective in practice because, it makes covnets invariant to variations in the presentation of an image and it also reduces the effects of background noise. Max Pooling has worked best in recent years, it is based on the idea that the maximum pixel in a region represents the most important feature in that region. Often images of objects we wish to classify could contain a number of other objects, for example, a cat appearing somewhere in the picture of a car could mislead the classifier. Pooling helps to alleviate the effects of this, and makes covnets generalize better.
It also greatly reduces the computational cost of the covnet. Generally, the size of images at each layer in the network is directly proportional to the computational cost (flops) of each layer. Pooling reduces the dimensions of the image as the layers get deeper, hence, it helps prevent an explosion of the number of flops a network requires. Strided convolutions are sometimes used as an alternative to pooling.
Overfitting is a phenomenon whereby a network works well on the training set, but performs poorly on the test set. This is often due to excessive dependence on the presence of specific features in the training set. Dropout is a technique for combating over-fitting. It works by randomly setting some activations to 0, essentially killing them. By doing this, the network is forced to explore more ways of classifying the images instead of over-depending on some features. This was one of the key elements in the AlexNet.
A major problem with neural networks is vanishing gradients. This is a situation whereby the gradients become too small, hence, training surfers terribly. Ioffe and Szegedy from Google Brain discovered that this was largely due to internal covariate shift, a situation that arises from the change data distribution as information flows through the network. What they did was to device the technique known as batch normalization. This works by normalizing every batch of the image to have zero mean and unit variance.
It is usually placed before non-linearity(relu) in cnns. It greatly improves accuracy while incredibly speeding up the training process.
The last ingredient required or modern covnets is data augmentation. The human vision system is excellent at adapting to image translations, rotations and other forms of distortions. Take an image and flip it anyway, most people can still recognize it. However, covnets are not very good at handling such distortions, they could fail terribly due to minor translations. They key to resolving this is to randomly distort the training images, using horizontal flipping, vertical flipping, rotation, whitening, shifting and other distortions. This would enable covnets to learn how to handle this distortions, hence, they would be able to work well in the real world.
Another common technique is to subtract the mean image from every image and also divide by the standard deviation.
Having bored you with the explanations of what these components are and why they work well, I shall now explain how to implement them in keras.
In this post, all experiments would be on CIFAR10, a data set of 60,000 32 x 32 RGB images. It is divided into 50,000 training images images and 10, 000 test images
To make things more modular, let’s create a simple function for each layer
def Unit(x,filters):    out = BatchNormalization()(x)    out = Activation("relu")(out)    out = Conv2D(filters=filters, kernel_size=[3, 3], strides=[1, 1], padding="same")(out)    return out
Here is the most vital aspect of our code, the Unit function defines a simple layer that contains three layers, first is BatchNormalization which i earlier explained, next we add RELU activation and finally, we add the convolution, notice how i put RELU before conv, it is a recent practice called “Pre-Activation”
Now we shall combine many of this unit layer into a single model
def MiniModel(input_shape):    images = Input(input_shape)    net = Unit(images,64)    net = Unit(net,64)    net = Unit(net,64)    net = MaxPooling2D(pool_size=(2,2))(net)    net = Unit(net,128)    net = Unit(net,128)    net = Unit(net,128)    net = MaxPooling2D(pool_size=(2, 2))(net)    net = Unit(net,256)    net = Unit(net,256)    net = Unit(net,256)    net = Dropout(0.5)(net)    net = AveragePooling2D(pool_size=(8,8))(net)    net = Flatten()(net)    net = Dense(units=10,activation="softmax")(net)    model = Model(inputs=images,outputs=net)    return model
Here we use the functional API to define our model, we begin with three Unit cells with 64 filters each, a Max Pooling layer follows, reducing our 32 x 32 images to 16 by 16. Next is 3, 128 filters Units followed by pooling, at this point, our images become 8 x 8, finally, we have another 3 units with 256 channels. Notice that each time we reduce our image dimensions by a factor of 2, we double the number of channels.
We add dropout with a ratio of 0.5, this would randomly deactivate 50% of our parameters, as i earlier explained, it combats overfitting.
Next we need to load the cifar10 dataset and perform some data augmentation
#load the cifar10 dataset(train_x, train_y) , (test_x, test_y) = cifar10.load_data()#normalize the datatrain_x = train_x.astype('float32') / 255test_x = test_x.astype('float32') / 255#Subtract the mean image from both train and test settrain_x = train_x - train_x.mean()test_x = test_x - test_x.mean()#Divide by the standard deviationtrain_x = train_x / train_x.std(axis=0)test_x = test_x / test_x.std(axis=0)
In the code above, after loading the train and test data, we subtract the mean image from each image and divide by the standard deviation, this is a basic data augmentation technique, sometimes, we might subtract the mean only and skip the standard deviation part, whichever works best should be used.
For more advance data augmentation, our image loading process would slighly change, keras has a very useful Data Augmentation utility that simplifies the whole process.
The code below would do the trick
datagen = ImageDataGenerator(rotation_range=10,                             width_shift_range=5. / 32,                             height_shift_range=5. / 32,                             horizontal_flip=True)# Compute quantities required for featurewise normalization# (std, mean, and principal components if ZCA whitening is applied).datagen.fit(train_x)
In the above, first, we specify a rotation angle of 10 degrees, a shift of 5/32 for both height and width and finally horizontal flip, all these transformations would be randomly applied to the images in the training set. Note that many more transformations exist, you can take a look at all parameters you can specify for the class. Bear in mind that over use of data augmentation might be detrimental.
Next, we have to convert the labels to one-hot encoding
#Encode the labels to vectorstrain_y = keras.utils.to_categorical(train_y,10)test_y = keras.utils.to_categorical(test_y,10)
I already explained this in my previous tutorials, so I won’t be explaining them here again. Infact, almost everything else that makes up the training process is exactly as in my previous tutorials, hence, here is the full code
#import needed classesimport kerasfrom keras.datasets import cifar10from keras.layers import Dense,Conv2D,MaxPooling2D,Flatten,AveragePooling2D,Dropout,BatchNormalization,Activationfrom keras.models import Model,Inputfrom keras.optimizers import Adamfrom keras.callbacks import LearningRateSchedulerfrom keras.callbacks import ModelCheckpointfrom math import ceilimport osfrom keras.preprocessing.image import ImageDataGeneratordef Unit(x,filters):    out = BatchNormalization()(x)    out = Activation("relu")(out)    out = Conv2D(filters=filters, kernel_size=[3, 3], strides=[1, 1], padding="same")(out)    return out#Define the modeldef MiniModel(input_shape):    images = Input(input_shape)    net = Unit(images,64)    net = Unit(net,64)    net = Unit(net,64)    net = MaxPooling2D(pool_size=(2,2))(net)    net = Unit(net,128)    net = Unit(net,128)    net = Unit(net,128)    net = MaxPooling2D(pool_size=(2, 2))(net)    net = Unit(net,256)    net = Unit(net,256)    net = Unit(net,256)    net = Dropout(0.25)(net)    net = AveragePooling2D(pool_size=(8,8))(net)    net = Flatten()(net)    net = Dense(units=10,activation="softmax")(net)    model = Model(inputs=images,outputs=net)    return model#load the cifar10 dataset(train_x, train_y) , (test_x, test_y) = cifar10.load_data()#normalize the datatrain_x = train_x.astype('float32') / 255test_x = test_x.astype('float32') / 255#Subtract the mean image from both train and test settrain_x = train_x - train_x.mean()test_x = test_x - test_x.mean()#Divide by the standard deviationtrain_x = train_x / train_x.std(axis=0)test_x = test_x / test_x.std(axis=0)datagen = ImageDataGenerator(rotation_range=10,                             width_shift_range=5. / 32,                             height_shift_range=5. / 32,                             horizontal_flip=True)# Compute quantities required for featurewise normalization# (std, mean, and principal components if ZCA whitening is applied).datagen.fit(train_x)#Encode the labels to vectorstrain_y = keras.utils.to_categorical(train_y,10)test_y = keras.utils.to_categorical(test_y,10)#define a common unitinput_shape = (32,32,3)model = MiniModel(input_shape)#Print a Summary of the modelmodel.summary()#Specify the training componentsmodel.compile(optimizer=Adam(0.001),loss="categorical_crossentropy",metrics=["accuracy"])epochs = 20steps_per_epoch = ceil(50000/128)# Fit the model on the batches generated by datagen.flow().model.fit_generator(datagen.flow(train_x, train_y, batch_size=128),                    validation_data=[test_x,test_y],                    epochs=epochs,steps_per_epoch=steps_per_epoch, verbose=1, workers=4)#Evaluate the accuracy of the test datasetaccuracy = model.evaluate(x=test_x,y=test_y,batch_size=128)model.save("cifar10model.h5")
A few things are different here, first
input_shape = (32,32,3)model = MiniModel(input_shape)#Print a Summary of the modelmodel.summary()
As i earlier explained, cifar 10 is made up of 32 x 32 RGB images, hence, the input shape has 3 channels. This is quite self-explanatory.
The next line creates an instance of the model we defined and we pass in the input shape
Finally, the last line would print out a full summary of our network including the number of parameters.
The last part that needs explaining is
epochs = 20steps_per_epoch = ceil(50000/128)# Fit the model on the batches generated by datagen.flow().model.fit_generator(datagen.flow(train_x, train_y, batch_size=128),                    validation_data=[test_x,test_y],                    epochs=epochs,steps_per_epoch=steps_per_epoch, verbose=1, workers=4)#Evaluate the accuracy of the test datasetaccuracy = model.evaluate(x=test_x,y=test_y,batch_size=128)model.save("cifar10model.h5")
First we define the number of epochs to run and also, the number of steps per epoch, don’t get confused with the numbers
steps_per_epoch = ceil(50000/128)
50000 here is the number of training images in total, here we are using a batch size of 128, that means, for each of the 20 epochs, the network would have to go over 50000/128 batches of images.
Next is the fit function, this is clearly different from the fit function i explained in my earlier tutorials.
A second look below would help
Fit the model on the batches generated by datagen.flow().model.fit_generator(datagen.flow(train_x, train_y, batch_size=128),                    validation_data=[test_x,test_y],                    epochs=epochs,steps_per_epoch=steps_per_epoch, verbose=1, workers=4)
Due to our use of the data generator class for data augmentation purposes, we have to use the fit_generator function, also we do not pass in train_x and train_y directly, instead we pass them via the flow function from the data generator, we also specify the batch size, next we state the validation data which in this case is the test data. All other things remain same.
This setup yields 82% after 20 epochs.
You can try tweaking the parameters and the network to see how much you can improve the accuracy. In the next tutorial i would be explaining some other tricks and techniques required to truly build very efficient cnn architectures. The purpose of this tutorial is to introduce you to the basic components.
If you want to get deeper into computer vision. download my free ebook “Introduction to Deep Computer Vision” from https://john.specpal.science
If you have any questions, comment below or reach to me on twitter via @johnolafenwa