## 2 Background: Deep Neural Networks
### Layers  
As described in the previous paragraphs, neurons are organized in layers that can have different shapes and characteristics. This section presents a short overview of different layers that are most commonly used in NNs.  
Fully Connected (FC) Layers. In FC layers, the neurons are arranged in the shape of a vector (see Figure 4). Considering a layer with \(C_{o}\) neurons and \(C_{i}\) inputs, each neuron \(c_{o}\) receives all the \(C_{i}\) inputs (Equation (2)). Therefore, each neuron has \(C_{i}\) weights and the total number of weights of the layer is \(C_{i}\times C_{o}\).  
\[\begin{split}\  
Convolutional (Conv) Layers. The inputs and outputs of a Conv layer are organized in 2D grids, defined as feature maps (FM). Multiple feature maps can form each input/output: the number of input/output feature maps is referred to as the number of input/output channels. The neurons of the Conv layer, rather than analyzing the whole input, receive only a sub-grid of dimension \(H_{k}\times W_{k}\) (\(C_{i}\times H_{k}\times W_{k}\) if there are multiple input channels). Horizontally adjacent neurons process grids of adjacent inputs separated by \(S\) positions, where \(S\) is a parameter known as stride. As shown in Figure 5, neurons that produce values belonging to the same output feature map (OFM) usually share the same kernel of weights. Therefore, each neuron of OFM \(c_{o}\) has a kernel of \(C_{i}\times H_{k}\times W_{k}\) weights, and the total number of weights is \(C_{o}\times C_{i}\times H_{k}\times W_{k}\). Equation (3) describes the operations performed in the Conv layer.  
\[\begin{split}\textbf{Ofm}[c_{o},h_{o},w_{o}]=\sum_{c_{i}=1}^{C_{ i}-1}\sum_{h_{k}=0}^{H_{k}-1}\sum_{w_{k}=0}^{W_{k}-1}\textbf{W}[c_{i},c_{o},h_{k},w_{ k}]\textbf{Ifm}[c_{i},Sh_{o}+h_{k},Sw_{o}+w_{k}]+\textbf{b}[c_{o}]\\ 0\leq c_{o}<C_{o},\ \ 0\leq h_{o}<H_{o},\ \ 0\leq w_{o}<W_{o}\\ 0\leq h_{k}<H_{k},\ \ 0\leq w_{k}<W_{k}\end{split} \tag{3}\]  
Normalization Layers. Batch Normalization (BN) layers are often inserted at various points in the neural networks after Conv or FC layers. As can be seen from Equation (4) describing the BN layers, the values are processed so that their mean is zero, and the variance is 1. \(\gamma\) and \(\beta\) are two trainable parameters inserted to integrate normalization in the training phase.  
\[y=\frac{x-E[x]}{Var[x]+\epsilon}\cdot\gamma+\beta \tag{4}\]  
The BN layers have two primary purposes. They contribute to accelerating the convergence of the training phase. Since the values always maintain a constant distribution, the network does not  
Figure 4: A fully connected layer with \(C_{i}=5\) and \(C_{o}=4\).  
have to adapt to different ranges at each training step. Moreover, they avoid value saturation of the values inside the network. Since saturating non-linear functions, such as Sigmoid, Tanh, or Softmax, are often used, having values with zero mean and variance 1 prevents too many values from being saturated, which would cause a considerable loss of information and slow down the training process. Pooling Layers. The main purpose of the pooling layers is to shrink the size of the feature maps within the network, to decrease the number of parameters and the number of operations to be performed. In the pooling layers, different sub-grids of the input feature maps are selected. For each sub-grid, a single value is calculated, which is a statistical metric of the group, e.g., the maximum value (MaxPooling) or the average value (AvgPooling). The sub-grids are usually selected of equal size, adjacent and non-overlapping.