“A. Convolution
Let’s assume that the input of our neural network has the presented shape in Fig. 2, It can be an image (e.g. color image of a CIFAR-10 dataset with a width and height of 32×32 pixels, and a depth of 3 which RGB channel) or a video (gray scale video whose height and width are the resolution, and the depth are the frames) or even an experimental video, which has width and height of (L×L) sensor values, and the depths are associated with different time frames, as in [2], [10], [15].

Fig. 2. - Three dimensional input representation of cnn

Why convolution ?. Let’s assume that the network receives raw pixels as input. Therefore, to connect the input layer to only one neuron (e.g. in the hidden layer in the Multi- Layer perceptron), as an example, there should be 32×32×3 weight connections for the CIFAR-10 dataset.

If we add one more neuron into the hidden layer, then we will need another 32×32×3 weight connection, which will become in total, 32×32×3×2 parameters. To make it clearer, more than 6000 weight parameters are used to connect the input to just only two nodes. It may be thought that two neurons might not be enough for any useful processing for an image classification application. To make it more efficient, we can connect the input image to the neurons in the next layer with exactly the same values for the height and width. It can be assumed this network is applied for the type of processing such as the edge in the image. However, the mentioned network needs 32×32×3 by 32×32 weight connections, which are (3,145,728), as in [4,5,14].

Therefore, looking for a more efficient method, it emerged that instead of a full connection, it is a good idea to look for local regions in the picture instead of in the whole image. Fig. 3, shows a regional connection for the next layer. In other words, the hidden neurons in the next layer only get inputs from the corresponding part of the previous layer. For example, it can only be connected to 5×5 neurons. Thus, if we want to have 32×32 neurons in the next layer, then we will have 5×5×3 by 32×32 connections which is 76,800 connections (compared to 3,145,728 for full connectivity), as in [1,9,13,14,17].

Fig. 3. - Convolution as alternative for fully connected network.

Although the size of connection drastically dropped, it still leaves so many parameters to solve. Another assumption for simplification, is to keep the local connection weights fixed for the entire neurons of the next layer. This will connect the neighbor neurons in the next layer with exactly the same weight to the local region of the previous layer. Therefore, it again drops many extra parameters, and reduces the number of weights to only 5×5×3=75 to connect 32×32×3 neurons to 32×32 in the next layer [5], [8], [11].

There are many benefits to these simple assumptions. Firstly, the number of connections decrease from around 3 million to only 75 connections in the presented example. Secondly, and a more interesting concept, is that fixing the weights for the local connections is similar to sliding a window of 5×5×3 in the input neurons and mapping the generated output to the corresponding place. It provides an opportunity to detect and recognize features regardless of their positions in the image. This is the reason why they are called convolutions [6,7,16].

To show the astounding effect of the convolution matrix, Fig.4, depicts what will happen if we manually pick the connection weight in a 3×3 window.

As we can see Fig. 4, the matrix can be set to detect edges in the image. These matrices are also called a filter because they act like the classic filters in the image processing. However, in the convolutional neural network these filters are initialized, followed by the training procedure shape filters, which are more suitable for the given task.

Fig. 4. - Effects of different convolutio matrix.

To make this method more beneficial, it is possible to add more layers after the input layer. Each layer can be associated with different filters. Therefore, we can extract different features from the given image. Fig. 5, shows how they are connected to the different layers. Each layer has its own filter and therefore extracts different features from the input. The neurons shown in Fig. 5, use a different filter, but look at the same part of the input image. [6], [8], [15], [17]–

Fig. 5. - Multiple layers which each of them correspond to different filter but looking at the same region in the given image

Multiple layers which each of them correspond to different filter but looking at the same region in the given image”
From the above paper, explain and summarize the principle of Convolution with some math provided above in detail. Mention all the Fig. above for the explanation. Try to use your own words, do not directly copy from the above sentences.