## 2 Background Theory
### Convolutional Networks  
Convolutional neural network (CNN) is a type of deep network that uses convolutional and pooling operation to extract the topological features of the input data. CNN is primarily used to solve difficult image-driven pattern recognition tasks, and if trained well, it will learn the features of the image completely. Therefore, in recent years, CNN has been widely used in image pattern recognition and image classification. CNN architectures come in several variations; however, a typical CNN includes convolutional layers, pooling layers, and fully connected layers. In the convolutional layer, a features map of the previous layer is convolved with multiple filters (also called Kernel) and is sent to the activation function to construct the output features map [29]:  
\[x_{j}^{l}=f\left(\sum_{i\in M_{j}}x_{i}^{l-1}*k_{ij}^{l}+b_{j}^{l}\right) \tag{5}\]  
where \(x_{i}\) is the \(i\) th input feature map, \(*\) stands for the convolutional operator, \(K\) denotes a \(w\times w\) convolutional filter, \(b_{j}\) is an additive bias, \(M_{j}\) is a set of input feature map, \(l\) is the \(l\)th layer in the network, and \(f(\cdot)\) is a nonlinear activation function. The mathematical inverse of the convolutional layer used in the decoder is known as the deconvolution layer. Different nonlinear functions such as a rectified linear unit (ReLU), sigmoid function, and scaled exponential linear unit (SELU) function can be used in convolutional layers. SELU is a variant of the ReLU activation function that, due to its self-normalizing properties, makes learning highly robust and allows training of networks which have many layers. SELU, ReLU, and sigmoid activation functions are defined as  
\[SELU(x)\ =\ \lambda\left\{\begin{array}{cc}x,&if\ x\geq 0\\ \alpha e^{x}-\alpha,&if\ x<0\end{array}\right. \tag{6}\]  
\[ReLU\ (x)\ =\max(0,x) \tag{7}\]  
\[\mathrm{sigmoid(x)\ =\frac{1}{1+e^{-x}}} \tag{8}\]  
For standard scale inputs (zero mean and standard deviation), the selected values for the parameters are \(\alpha\approx 1.6732\) and \(\lambda\approx 1.0507\)[40].  
The pooling layer usually follows the convolutional layer and is used to reduce the computational load by reducing the size of the features map. Two common pooling methods are max pooling and average pooling, which perform local max and average operations over the input features, respectively. The calculation process of the pooling layer is given as [29]  
\[X_{j}^{l}=f\left(\beta_{j}^{l}\textit{down}\left(X_{j}^{l-1}\right)+b_{j}^{l}\right) \tag{9}\]  
where \(\beta_{j}\) is the weight of pooling and \(b_{j}\) is the additive bias, and \(\textit{down}(x)\) denotes the down-sampling function, e.g., max pooling. In contrast to the pooling layer, an upsampling layer is a simple layer in the decoder with no weights that is used to increase the dimensions of input.  
In a fully connected layer, the features maps are converted into a one-dimensional feature vector, and all neurons of both layers are connected, like a traditional multilayer neural network. The output of the fully connected layer can be obtained as [29]  
\[O_{j}=f\left(\sum_{j=1}^{d}x_{j}^{F}\beta_{j}^{l}+b_{j}^{l}\right) \tag{10}\]  
where \(O_{j}\) is the output value; \(x_{j}\) is the \(j\)th neuron in the fully connected layer; \(\beta_{j}\) and \(b_{j}\) are the weight and the additive biases corresponding to \(O\) and \(x_{j}\), respectively; and \(f(\cdot)\) is an activation function.