## 2 Methodology
### Deep Neural Networks' Setup  
Convolutional neural networks are deep neural networks specifically tailored for analyzing visual imagery. The major advantage of ConvNets is that they require little preprocessing compared to other image classification algorithms. Three main types of layers that make up ConvNet architectures are convolutional layers, pooling layers, and fully connected layers. The main building block of ConvNets is the convolutional layer. Figure 7: The original training sample (left, framed) and its transformations that can potentially be used for data augmentation. A convolution is the application of a filter to the layer input that results in a map of activations (feature map), indicating the locations and strength of a detected feature in an input [74]. The convolution is performed by sliding a \(K\times K\) convolution filter (kernel) over the input image with a predetermined step size (stride). The innovation of using the convolution operation in a neural network is that the values of the filter are learned during the training of the network. Under stochastic gradient descent, the network is forced to learn to extract features from the input that are most useful for classifying images. ConvNets usually learn multiple (32-512) filters in parallel for a given input. A filter must have the same number of channels (depth) as the input and can have specific filter values for each of the input channels. Regardless of the depth of the input and depth of the filter, each filter produces a 2D feature map because eventually, the channels are summed together to form one single channel (element-wise addition). The Rectified Linear Unit (ReLU) is a supplementary step to the convolution operation. Its purpose is to increase the non-linearity in feature maps. The result of the convolution operation is passed through the ReLU activation function so the values in final feature maps are not just the sums, but the ReLU function applied to the sums. The ReLU activation function has rapidly become the default activation function for most types of neural networks. It provides true zero and acts like a linear function for the most part, but is actually a nonlinear function allowing complex relationships in the data to be learned. ReLU is also easy to implement, and networks trained with this activation function avoid the problem of vanishing gradients [75]. The depth of the output of a convolutional layer is determined by the number of filters because each of them creates a distinct feature map. The width and height of the output of a convolutional layer are, on the other hand, determined by the formula:  
\[D_{o}=1+\frac{D_{i}-K+2P}{S}, \tag{5}\]  
where \(D_{o}\) and \(D_{i}\) are the height/length of the output and input, \(S\) is the stride, and \(P\) is the width of the added border of zeros (zero-padding). Note that commonly, \(K=3,P=1,S=1\), and \(D_{o}=D_{i}\). Pooling layers do not affect the depth dimension, but perform a downsampling operation along the spatial dimensions (width, height) of the input for the next convolutional layer. The decrease in size leads to less computational overhead for the upcoming layers of the network, works against over-fitting, and improves local translation invariance. Much like the convolution operation, the pooling layer takes a sliding window that is moved in stride across the input and transforms its values into a more representative value by selecting, e.g., the maximum value from the window (max pooling). Contrary to the convolution operation, however, pooling has no trainable parameters, although window (kernel) size and stride must be specified. Commonly, \(K=2,S=2\). Fully-connected layers are ordinary neural network layers that are fully connected with the output of the previous layer and are typically used in the last stages of the ConvNet. They are also used to construct the desired number of nodes in the output layer.