## 5 Deep Learning
### Elements of a CNN
#### Activation Layer  
The activation layers, also known as non-linearity layers, are crucial for enhancing the performance of CNN. As real data are non-linear, it is necessary to introduce non-linearityin the dataset. The activation layers are usually introduced after each convolution. There are several functions that are used to introduce this non-linearity, but maybe the most important ones are ReLU where the negative pixel values are replaced by zero, and sigmoid that usually returns a value between 0 and 1.  
Pooling Layer  
A pooling layer, used for sub-sampling, usually comes after the activation layer. Without sub-sampling, the volume of data would grow up continuously convolution after convolution, requiring a huge computational effort and, consequently, much more processing time. Sub-sampling aims to perform a filtering operation throughout the input data (the data coming from the corresponding activation layer) in order to eliminate a large part while maintaining the most relevant information. Therefore, only the relevant information will feed the next convolutional layer. This operation also normalises the data, which means that the variance of the data is reduced. In these layers, as in the convolutional ones, the stride and the size of the kernel or filter must be determined. A typical value of two is usually adopted for both parameters. There are different pooling or sub-sampling methods. The most relevant ones are briefly exposed below:  
1. **Average pooling**: offers the mean value of the sub-sampled pixels as the output value.
2. **Max pooling**: offers the highest value of the sub-sampled pixels as the output value.
3. **Other methods**: are not as popular as the previous ones. The reason is that they are more specific methods that offer a great performance under certain particular scenarios. Some examples are mixed pooling, stochastic pooling, spatial pyramid pooling (SPP) or region of interest pooling (ROIP).  
Figure 4 shows a graphical explanation of the operation of the max and average pooling algorithms. In this case, a \(2\times 2\) kernel and a stride with a value equal to 2 is used. It can be appreciated that max pooling provides the highest value of the kernel window, while average pooling yields the average value of the kernel window.  
The fully connected (FC) layers are connected to each of the neurons of the previous layer. These layers perform a high-reasoning operation, reducing the multidimensional data of the previous layers to a (\(1\times 1\times\) N) vector, where N is the number of output classes. Neurons in these layers also have an associated weight, which, as with the convolutional layer neurons, is adjusted after several iterations by the backpropagation algorithm. To avoid overfitting and to boost network performance, the dropout algorithm is included in these layers. The purpose of the algorithm is to eliminate the connections of the FC layer  
Figure 4: Max pooling and average pooling algorithms.  
with neurons whose weight is equal to zero, so that neurons having a zero contribution to the determination of a certain output class do not participate in the calculation. This reduces training time and contributes to a lighter system.  
The 1D vector obtained from the FC layers feeds the classification layer. The classification layer transforms the vector elements into probabilities of belonging to each class. The most common function in the classification layer is softmax. The probabilities obtained can be used directly to perform the classification, approximating the highest probability to 1 (the element belongs to that class) and the rest of the probabilities to 0 (the element does not belong to the rest of the classes) [52]. Another approach is to use an ML classifier such as the ones explained above to make the final classification. These classifiers may substitute the softmax function and also be complementary to it. Several classifiers can even be applied in parallel to compare results and receive feedback to help improve the reliability of the results.