Visualizing the Fundamentals of Convolutional Neural Networks | by Mark C. F. Sousa | Towards Data Science

Convolutional Neural Networks (CNNs) are a subtype of Artificial Neural Networks (ANNs) mostly used for image classification. CNNs follow the biological principle of the replication of a structure capable of identifying patterns to identify these patterns in different locations. It was inspired by the model of cats’ visual system proposed by the Nobel prizes winners Hubel and Wiesel at “Receptive fields, binocular interaction and functional architecture in the cat’s visual cortex”, published in 1962. One of the works that used this inspiration was the Fukushima’s Neocognitron, in 1980, although the word Convolution was not used at the time. Therefore, it is not a coincidence that CNNs are very successful in image recognition. However, they have also shown good results dealing with temporal data such as time series and speech recognition, or even when applied on graphs. The CNNs became very popular after winning the 2012 edition of the competition Imagenet Large Scale Visual Recognition Challenge (ILSVRC) by a margin of roughly 10%. Alex Krizhevsky and Ilya Sutskever, under the guidance of Geoffrey Hinton, submitted the CNN architecture that became famous under the name of “AlexNet”. At the time, Geoffrey Hinton had already made significant scientific contributions to the field of ANNs. He was one of the contributors of the Backpropagation algorithm in 1986, and Boltzmann Machines in 1983. These are some of the reasons that Geoffrey Hinton was recognised as one of the fathers of Deep Learning.
A typical CNN is composed by a series of convolutional layers, that act as features extractors, followed by a classifier, usually a Multilayer Perceptron (MLP), also known as Fully Connected Layers (FC layers), as shown by Figure 1.
The first layer receives the input image represented in three colour channels, the RGB channels. Then, the first layer executes the convolutions of the input image with multiple kernels, resulting in a set of feature maps of the first layer. Each feature map determines the intensity and location of a specific feature. The feature map extracted by a convolutional layer can be submitted to a downsampling operation known as Pooling. The Pooling operation is optional, so it may not follow every convolutional layer. The result of a Pooling layer is another set of feature maps, with the same number of maps, but with reduced resolution. The following convolutional layer uses the feature map from the previous layer to execute more convolutions and generate new feature maps. The feature maps from the last layers are the input of the classifier, the FC layers.
The convolution operation, which is denoted by an ∗ (asterisk), can be described as:
being x some type of input, such as a sensor signal, t a given time, and k the kernel applied.
One important property of the convolution operation is that it is commutative, which means that (x∗ k)=( k∗x) as follows:
On the other hand, the cross-correlation operation, denoted by ⋆ (five-pointed star), is not commutative and can be described as:
The commutative property of the convolution emerges from the fact that the kernel is flipped relative to the input. The flip occurs as a result of index manipulation. Note that the index for the input x is a and the index for kernel is t−a. Even though the commutative is a valuable property for writing mathematical proofs, it is not as relevant for neural network implementation. In fact, many machine learning libraries implement the cross-correlation instead of convolution and refer to both operations as convolution. As a consequence, the kernel that is learned during training will be flipped in comparison to a library that actually implements the convolution as described by equation 1. We will follow the same convention in this text and call the cross-correlation convolution.
We can adapt the equation 3 for the convolution with 2D data, such as grey-scale images:
being r[i,j] a discrete output of the convolution, h the height of the kernel, w the width of the kernel, x[a,b] the patch of the grey-scale image, and k[i+a, j+b] the kernel.
In other words, the convolution operation extracts multiple patches of pixels from the image to multiply by the kernel. The kernel is basically a matrix of weights. The patch of pixels extracted from the image is often known as receptive field — in biology, the receptive field is a sensorial region that stimulates a neuron. The multiplication between the receptive field and the kernel consists of an element-wise multiplication between each pixel and the respective element of the kernel. After the multiplications, the results are added to form one element of the feature map, defined in equation 4 by r[i,j].
The following animation shows the convolution operation between a 5x5 grey-scale image and a 3x3 kernel. The receptive field is highlighted by the red square. The output of this convolution is a 3x3 feature map.
The actual image used in the animation can be seen in Figure 3 below. The values in the kernel and the feature map were rescaled to fit into the interval between 0 and 255 to be represented as grey-scale pixels. The brighter pixels in the image represent higher values from convolution, while the darker pixels represent lower values.
The convolution above uses kernel 3x3, consequently, there are nine possible receptive fields in the input, each with size 3x3. Note that the receptive fields composed of mostly white pixels or composed of mostly dark pixels result in a very dark pixel after the convolution. On the other hand, the receptive fields that are composed by three bright pixels on the left, intermediate pixels in the middle and dark pixels on the right, result in the brightest pixels after the convolution. This is because this type of kernel is useful to highlight edges, specifically edges transitioning from a bright region on the left to a dark region on the right.
Now, see what happens when we apply the same kernel to an image that also contains the opposite transition, from a dark region on the left to a bright region on the right. In Figure 4, the receptive fields that exhibit the transition from dark to bright resulted in the darkest pixels. Note that the former transition (from bright to dark) still resulted in the brighter pixels. It means that this kernel not only detects the edges transitioning from bright to dark but also detects the opposite edges, from dark to bright. One type of edge results in the most positive values, while the other type of edge results in the most negative values.
The convolution for RGB images is quite similar to the grey-scale case. The equation 4 can be adapted to RGB image adding another loop to iterate over the RGB channels as follows:
The additional loop over the variable c allows the iteration on the channels RBG. As a result, the sum is done over three-dimensional data, instead of bi-dimensional, and still results in a single value for each three-dimensional receptive field and kernel.
Let’s start this topic with a practical example. Look at the result of the following three convolutions in Figure 5. In order to illustrate the result of the convolutions, each of the three kernels in the following examples consists of a small patch extracted from the image.
In the first example, the patch composing the kernel comprises the region of the plane with the number six in white. The grey-scale image on the right is essentially the result of the convolution between the kernel and the image. The darkest pixels represent the smallest results of the operation between a receptive field and the kernel, on the other hand, the brightest pixels represent the highest values for the operation between a receptive field and the kernel.
In the second example, the kernel consists of the patch of pixels forming the wheel of the plane. In the third example, the kernel consists of a patch of yellow pixels copied from the tail of the plane.
Note that the brightest pixels in each resulting image correspond to the location that originated each kernel. In the first example, it corresponds to the location of the number 6. In the second example, it corresponds to the location of the wheels. Even though the kernel is a copy of one of the wheels, the other wheel is quite similar and also resulted in bright pixels. In the third example, the brightest pixels correspond to all the yellow region of the plane.
The stride is the distance between each receptive field. All the examples we have shown so far use a stride of one. The adoption of such small strides results in a big overlap between receptive fields. As a result, a lot of information is repeated in adjacent receptive fields, as shown in Figure 6.
In the case of a kernel with dimensions 3x3, the adoption of a stride of 2 results in one column or one row overlapping with adjacent receptive fields. This overlap is desired to guarantee that the stride doesn’t skip important information.
Increasing the stride will reduce the computational cost of the convolutions. If we change the stride from one to two, the reduction in the computational cost is about four. It happens because the stride affects the distance between receptive fields in both dimensions. Similarly, if we triple the stride, we can expect a computational cost reduction of roughly nine times. The computation cost is reduced because the increase in stride reduces the number of receptive fields extracted from the input, consequently, the dimension of the output is also reduced.
Figure 7 shows four results of convolution using strides of 2, 4, 8, and 16. The kernel size used in the convolutions is 70x70. Note that increasing the stride by a factor of two, the execution time is reduced by almost four.
In Figure 7, the result of the convolution using a stride of 16 has four times fewer pixels than the result using the stride of 8. Note that adopting the stride of 16 results in an overlapping of 54 rows or columns, because the receptive field size is 70x70. With the stride of 16, it is still possible to identify the highest values of the convolution by the brightest pixels in the wheel of the plane.
In this section, we will study how the forward propagation works in the convolutional layers. To do so, we will understand how a single convolutional layer works and then understand how multiple layers work together. In this study, we will learn two new concepts: non-linear activation and pooling operation.
Figure 8 shows the forward propagation in a typical convolutional layer, which consists of three stages: convolutions, non-linear activations, and pooling. The convolution operation was already discussed in the first section. Now, we will see the other two operations.
The non-linear activation is also known as the detector stage. In this stage, the result of the convolutions and the bias are submitted to a non-linear activation such as the ReLU function. The non-linear activation doesn’t change the dimensions of the feature map, it only modulates the values in it.
First, what is linear activation? A linear activation is a function that follows the rule f(x)=ax, where a is a constant and x the variable. Its graph is a straight line through the origin (0, 0). It means that the functions in the shape f(x)=ax + b, in which a and b are constant, are not linear. Both are affine functions, but only the one with a single constant multiplying the variable is a linear function.
To a function be linear, when we multiply the input by a constant α, we should also see the output multiplied by the same constant α. That means:
Another requirement is that when we apply the sum of two inputs in a linear function, we should get the output equivalent of the sum of the two variable applied individually to the function:
Now, why we do we use non-linear activation? Because when we apply linear combinations (additions or multiplications) in linear functions, the result is also linear. Even though many models can be roughly approximated by linear models, using non-linearities in ANNs makes it capable of representing both linear and non-linear models. In other words, non-linearities make ANNs more powerful function approximators.
One of the most common non-linear activation used in deep learning is the ReLU function, which stands for Rectified Linear Unit. This function is given by:
This function, when applied with a bias, results in a graph illustrated in Figure 9.
Figure 10 shows how the ReLU modulates the result of the convolution. Here, the same results from Figure 2 are used to apply the ReLU function.
The equivalent image of the ReLU function used in the example above is shown in Figure 11. Note that some of the intermediate values were blacked-out, highlighting the brightest three pixels.
Next, in Figure 12, you can see some options of bias in the ReLU function. The function takes the result of the convolution of the plane with the patch of the image containing the wheel as input. Note that the bias behaves like a threshold that determines what is shown and what is not.
The threshold behaviour depicted in Figure 12 resembles the biological neurons, that doesn’t fire when it receives a stimulus under a certain threshold. If the stimulus surpasses the threshold, the neuron starts to fire, and also fires more frequently as the stimulus increases. In Figure 12, when the bias is 500, it is contributing to the activity of the artificial neuron. However, if we define the bias as -1000, the artificial neuron only fires with the strongest stimulus.
To conclude, the ReLU function works as an artificial neuron. That is why this phase is called the detection phase. The ReLU function is responsible for detecting the presence of the feature extracted by the kernel. Consequently, there is a single bias for each the kernel, because each feature requires a different threshold.
At last, but not least, the pooling operation. It is a downsampling operation executed over each feature map. It extracts receptive fields from the feature map and replaces it with a single value. This single value can be obtained by different aggregation criteria, such as maximum value, average, or weighted average according to the distance from the centre of the receptive field.
Besides the aggregation criteria, there are other two hyperparameters in the pooling operation, the size of the receptive field and the stride.
Similar to the stride, the pooling operation results in fewer data processed by the convolutions. One difference is that instead of skipping data, the pooling operation tries to summarise the receptive field into a single value. Another difference is that the stride is applied before the convolution, while pooling is applied over the result of a convolution, reducing the volume of data to the next layer. Furthermore, the receptive field of the pooling operation is bi-dimensional, because it is applied to each feature map individually, while the receptive field of the convolution is three-dimensional, comprising a section of all the feature maps in a layer.
One desired side-effect of the pooling operation is that is increases the invariance of the network to translations of the input. This effect is amplified as more convolutional layers are followed by pooling layers.
Figure 13 shows the propagation of values through two pooling layers with pooling size of 3x3 and stride of 2. Any activation in the region covered in blue the input feature map affects the region covered in blue in the outcome of pooling 1. Similarly, the activations in the region covered in blue in the outcome of pooling 1 affect the region covered in blue in the outcome of pooling 2. The same relationship is valid between the regions covered in green.
Considering that the pooling in Figure 13 is max pooling, it doesn’t matter where in the blue region the highest value occurs in the input feature map, because it will be propagated to the blue activation in the outcome pooling 2 in the same way. This is the reason why pooling layers enhance translation invariance, small translations in the input don’t change the values in the output.
Figure 14 shows the effect of different combinations of strides in convolution, biases in the ReLU function, pooling sizes, and pooling strides. On the left, there are three examples of strides: 2, 9, and 16. For each option of stride, there are three examples of bias: 500, -250, and -1000. For each bias, there are three examples of pooling size and stride: pooling size 3x3 and stride of 2, pooling size 5x5 and stride of 3, and pooling size 7x7 and stride of 4x4.
The effect of the stride in the convolution and the stride in the max pooling is cumulative. When we use a stride of two in both convolution and in the max pooling, the final result is a reduction of roughly four times in the width and in the height of the feature map.
The values 16 for stride in the convolution and four in the stride of max pooling are unusual. They were intentionally exaggerated to illustrate the impact they have in the final result of the convolutional layer. The resulting feature map has only 100 elements (10 x 10), much smaller than the feature map with 37636 elements (194 x 194).
We will come back to the AlexNet, the first famous CNN architecture. This is a good practical example to understand how the components of CNNs work together. The building blocks of AlexNet is represented in Figure 14. The dashed pyramids represent the execution of convolutions using a receptive field from the input or the feature map from the previous layer. The big boxes represent the feature maps and the small boxes inside the feature maps are the receptive fields. This CNN can classify objects in 1000 different classes.
One peculiarity of this architecture is that it was trained using two GPUs. The top elements of Figure 15 were allocated in one GPU, while the elements at the bottom were allocated in another GPU.
You can get this CNN already trained in some frameworks, such as PyTorch. It is relatively easy to use. You can also use transfer learning to apply this architecture to other image datasets. It requires a little data processing to normalize and rescale the images to match the requirements of AlexNet, but it worth the try. It would be something like:
from torchvision import transformspre_transforms = transforms.Compose([transforms.RandomResizedCrop(224),transforms.ToTensor(),transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])])
You can import the model in PyTorch as follows:
from torchvision import modelsmodel = models.alexnet(pretrained=True)
You can disable the training of the convolutional layers and create your own classifier to override the original AlexNet Fully Connected Layers:
from torch import nnfor param in model.parameters():    param.requires_grad = False # disable trainingfcLayersDict = OrderedDict([('fc1', nn.Linear(9216, 1024)),                   ('relu1', nn.ReLU()),                   ('fc2', nn.Linear(1024, 102)),                   ('output', nn.LogSoftmax(dim=1))                           ])model.fcLayersDict = fcLayersDictfc = nn.Sequential(fcLayersDict)model.classifier = fc
In addition to AlexNet, there are also other models in PyTorch that can be used to classify your own image dataset, given the necessary customizations. It is really recommended to test these models or even build your own CNN from scratch to understand how CNN works.
Even if you were already familiar with the concepts presented here, I hope you could at least see them from another perspective. There were some other concepts from CNNs that were not discussed here. If you are curious or are not sure about a particular aspect of CNNs, please, let me know.