"A. Convolution
Let’s assume that the input of our neural network has the
presented shape in Fig. 2, It can be an image (e.g. color image
of a CIFAR-10 dataset with a width and height of 32×32
pixels, and a depth of 3 which RGB channel) or a video (gray
scale video whose height and width are the resolution, and the
depth are the frames) or even an experimental video, which
has width and height of (L × L) sensor values, and the depths
are associated with different time frames, as in [2,10,15].
Why convolution ? . Let's assume that the network receives
raw pixels as input. Therefore, to connect the input layer to
only one neuron (e.g. in the hidden layer in the Multi- Layer
perceptron), as an example, there should be 32×32×3 weight
connections for the CIFAR-10 dataset.
Fig. 1. Learned features from a Convolutional Neural Network
Authorized licensed use limited to: Hong Kong Polytechnic University. Downloaded on April 08,2024 at 07:52:10 UTC from IEEE Xplore. Restrictions apply.
ICET2017, Antalya, Turkey
978-1-5386-1949-0/17/$31.00 ©2017 IEEE
Fig. 2. Three dimensional Input representation of CNN
If we add one more neuron into the hidden layer, then we will
need another 32×32×3 weight connection, which will become
in total, 32×32×3×2 parameters. To make it clearer, more than
6000 weight parameters are used to connect the input to just
only two nodes. It may be thought that two neurons might not
be enough for any useful processing for an image
classification application. To make it more efficient, we can
connect the input image to the neurons in the next layer with
exactly the same values for the height and width. It can be
assumed this network is applied for the type of processing
such as the edge in the image. However, the mentioned
network needs 32×32×3 by 32×32 weight connections, which
are (3,145,728), as in [4,5,14].
Therefore, looking for a more efficient method, it emerged
that instead of a full connection, it is a good idea to look for
local regions in the picture instead of in the whole image.
Fig. 3, shows a regional connection for the next layer. In other
words, the hidden neurons in the next layer only get inputs
from the corresponding part of the previous layer. For
example, it can only be connected to 5×5 neurons. Thus, if we
want to have 32×32 neurons in the next layer, then we will
have 5×5×3 by 32x32 connections which is 76,800
connections (compared to 3,145,728 for full connectivity), as
in [1,9,13,14,17].
Although the size of connection drastically dropped, it still
leaves so many parameters to solve. Another assumption for
simplification, is to keep the local connection weights fixed
for the entire neurons of the next layer. This will connect the
neighbor neurons in the next layer with exactly the same
weight to the local region of the previous layer. Therefore, it
again drops many extra parameters, and reduces the number of
weights to only 5×5×3=75 to connect 32×32×3 neurons to
32×32 in the next layer [5,8,11].
There are many benefits to these simple assumptions. Firstly,
the number of connections decrease from around 3
million to only 75 connections in the presented example.
Secondly, and a more interesting concept, is that fixing the
weights for the local connections is similar to sliding a
window of 5×5×3 in the input neurons and mapping the
generated output to the corresponding place. It provides an
opportunity to detect and recognize features regardless of their
positions in the image. This is the reason why they are called
convolutions [6,7,16].
Fig. 3. Convolution as alternative for fully connected network.
To show the astounding effect of the convolution matrix,
Fig.4, depicts what will happen if we manually pick the
connection weight in a 3×3 window.
As we can see Fig.4, the matrix can be set to detect edges
in the image. These matrices are also called a filter because
they act like the classic filters in the image processing.
However, in the convolutional neural network these filters are
initialized, followed by the training procedure shape filters,
which are more suitable for the given task.
To make this method more beneficial, it is possible to add
more layers after the input layer. Each layer can be associated
with different filters. Therefore, we can extract different
features from the given image. Fig. 5, shows how they are
connected to the different layers. Each layer has its own filter
and therefore extracts different features from the input. The
neurons shown in Fig. 5, use a different filter, but look at the
same part of the input image. [6,8,15,17]

Fig.4. Effects of different convolution matrix.
Authorized licensed use limited to: Hong Kong Polytechnic University. Downloaded on April 08,2024 at 07:52:10 UTC from IEEE Xplore. Restrictions apply.
ICET2017, Antalya, Turkey
978-1-5386-1949-0/17/$31.00 ©2017 IEEE

Fig. 5. Multiple layers which each of them correspond to different filter
but looking at the same region in the given image
B. Stride
 In fact, CNN has more options which provide a lot of
opportunities to even decrease the parameters more and more,
and at the same time reduce some of the side effects. One of
these options is stride. In the above mentioned example, it is
simply assumed that the next layer’s node has lots of overlaps
with their neighbors by looking at the regions. We can
manipulate the overlap by controlling the stride. Fig. 6, shows
a given 7×7 image. If we move the filter one node every time,
we can have a 5x5 output only. Note that the output of the
three left matrices in Fig. 6, have an overlap (and three middle
ones together and three right ones also). However, if we move
and make every stride 2, then the output will be 3x3. Put
simply, not only overlap, but also the size of the output will be
reduced. [5,12,16].
Equation (1), formalize this, given the image N×N
dimension and the filter size of the F×F, the output size O as
shown in Fig. 7.
 (1)
Where N is the input size, F is the filter size, and S is the
stride size.
C. Padding
One of the drawbacks of the convolution step is the loss of
information that might exist on the border of the image.
Because they are only captured when the filter slides, they
never have the chance to be seen. A very simple, yet efficient
method to resolve the issue, is to use zero-padding. The other
benefit of zero padding is to manage the output size. For
example, in Fig. 6, with N=7 and F=3 and stride 1, the output
will be 5×5 (which shrinks from a 7×7 input).
Fig. 7. The effect of stride in the output.
However, by adding one zero-padding, the output will be
7×7, which is exactly the same as the original input (The
actual N now becomes 9, use the formula (1). The modified
formula including zero-padding is formula (2).

 (2)
Where P is the number of the layers of the zero-padding
(e.g. P=1 in Fig. 8), This padding idea helps us to prevent
network output size from shrinking with depth. Therefore, it is
possible to have any number of deep convolutional networks.
[2,12].
D. Feature of CNNs
The weight sharing brings invariance translations to the
model. It helps to filter the learn feature regardless of the
spatial properties. By starting random values for the filters,
they will learn to detect the edge (such as in Fig. 4) if it
improves the performance. It is important to remember that if
we need to know that something is spatially important in the
given input, then it is an extremely bad idea to use a shared
weight.
Fig. 8. Zero-padding
Fig. 6. Stride 1, the filter window moves only one time for each connection
0 0 0 0 0 0 0 0 0
0 0
0 0
0 0
0 0
0 0
0 0
0 0
0 0 0 0 0 0 0 0 0
Authorized licensed use limited to: Hong Kong Polytechnic University. Downloaded on April 08,2024 at 07:52:10 UTC from IEEE Xplore. Restrictions apply.
ICET2017, Antalya, Turkey
978-1-5386-1949-0/17/$31.00 ©2017 IEEE
Figure (9). Visualizing Convolutional deep neural network layers
 This concept can be extended to different dimensions
also. For example, if it is sequential data such as an audio,
then it can employ a one dimensional audio. If it is an
image, as it is shown, two dimensional convolutions can be
applied. And for videos, or 3D images, a three dimensional
convolution can be used. This simple idea beat all the
classic object recognition methods in computer vision in the
2012 ImageNet challenge as shown in Fig. 9, [5,14,18]
E. Convolutional Formula
The convolution for one pixel in the next layer is
calculated according to the formula (3) .
Where is the output in the next layer, is the
input image and is the kernel or filter matrix and is the
convolution operation. Fig. 10, shows how the convolution
works. As can be seen, the element by element product of
the input and kernel is aggregated, and then represents the
corresponding point in the next layer. [4,9].
III. NONLINEARITY
The next layer after the convolution is non-linearity. The
non-linearity can be used to adjust or cut-off the generated
output. This layer is applied in order to saturate the output or
limiting the generated output.

Fig. 10. Details on Convolution layer
For many years, sigmoid and tanh were the most popular
non-linearity. Fig. 11, shows the common types of
nonlinearity. However, recently, the Rectified Linear Unit
(ReLU) has been used more often for the following reasons.
1. ReLU has simpler definitions in both function and
gradient.
 (4)
 (5)
2. The saturated function such as sigmoid and tanh
cause problems in the back propagation. As the
neural network design is deeper, the gradient signal
begins to
vanish, which is called the “vanishing gradient”.
This happens since the gradient of those functions is
very
close to zero almost everywhere but the center.
However, the ReLU has a constant gradient for the
positive input. Although the function is not
differentiable, it can be ignored in the actual
implementation.
3. The ReLU creates a sparser representation. because
the zero in the gradient leads to obtaining a
complete zero. However, sigmoid and tanh always
have non-zero results from the gradient, which
might not be in favor for training. [2,5,13].
Fig. 11. Common types of nonlinearity.
(3)
Authorized licensed use limited to: Hong Kong Polytechnic University. Downloaded on April 08,2024 at 07:52:10 UTC from IEEE Xplore. Restrictions apply.
ICET2017, Antalya, Turkey
978-1-5386-1949-0/17/$31.00 ©2017 IEEE
IV. POOLING
The main idea of pooling is down-sampling in order to
reduce the complexity for further layers. In the image
processing domain, it can be considered as similar to
reducing the resolution. Pooling does not affect the number
of filters. Max-pooling is one of the most common types of
pooling methods. It partitions the image to sub-region
rectangles, and it only returns the maximum value of the
inside of that sub-region. One of the most common sizes
used in max-pooling is 2×2. As can see in Fig. 12, when
pooling is performed in the top-left 2×2 blocks (pink area),
it moves 2 and focus on top-right part. This means that
stride 2 is used in pooling. To avoid down-sampling, stride
1 can be used, which is not common. It should be
considered that down-sampling does not preserve the
position of the information. Therefore, it should be applied
only when the presence of information is important (rather
than spatial information). Moreover, pooling can be used
with non-equal filters and strides to improve the efficiency.
For example, a 3x3 max-pooling with stride 2 keeps some
overlaps between the areas. [5,10,16].
V. FULLY-CONNECTED LAYER
The fully-connected layer is a similar to the way that neurons
are arranged in a traditional neural network. Therefore, each node
in a fully-connected layer is directly connected to every node in
both the previous and in the next layer as shown in Fig. 13, From
this figure we can note that each of the nodes in the last frames in
the pooling layer are connected as a vector to the first layer from
the fully-connected layer. These are the most parameters used with
the CNN within these layers, and take a long time in training [3,8].
The major drawback of a fully-connected layer, is that it
includes a lot of parameters that need complex computational in
training examples. Therefore, we try to eliminate the number of
nodes and connections. The removed nodes and connection can be
satisfied by using the dropout technique. For example, LeNet and
AlexNet designed a deep and wide network while keeping the
computational complex constant [4,6,9].
The essence of the CNN network, which is the
convolution, is when the nonlinearity and pooling layer are
introduced. Most common architecture uses three of them as"
From the above paper, explain and summarize the principle of CNN in detail with equations. Try to use your own words, do not directly copy from the above sentences.