# Congestive Heart Failure Category Classification Using Neural Networks in Short-Term Series
## 2 Models Based on Neural Networks
### Convolutional Neural Network  
There are numerous variants of CNN architectures in the state-of-the-art [57]. However, their basic components are three types of layers: convolutional, pooling, and fully connected layers [57; 58]. The convolutional layer learns meaningful representations of the inputs through a series of filters (or kernels). Each filter generates a feature map that defines the depth of the output feature cube. Specifically, each node in a feature map receivesinformation from a local region of neighborhood nodes in the previous layer. With a learned kernel and then applying an elementwise non-linear activation function, the activation value \(h^{l}_{i,j,k}\) of the \(k\)-th feature map of the \(l\)-th layer can be calculated as follows:  
\[h^{l}_{i,j,k}=\varphi(z^{l}_{i,j,k}), \tag{7}\]  
where \(\varphi(\cdot)\) denotes the non-linear function and \(z^{l}_{i,j,k}=\mathbf{w}^{l}_{k}\cdot\mathbf{x}^{l}_{i,j}+b^{l}_{k}\) is the feature value \((i,j)\) in the \(k\)-th feature map of the \(l\)-th layer. The weight vector \(\mathbf{w}^{l}_{k}\) and the bias term \(b^{l}_{k}\) of the \(k\)-th filter of the \(l\)-th layer are connecting links between the input patch \(\mathbf{x}^{l}_{i,j}\) centered at \((i,j)\) and the corresponding node in the \(k\)-th feature map. Note that the filter \(\mathbf{w}^{l}_{k}\) that generates the three-dimensional feature map \(\mathbf{z}^{l}_{i,\cdot,k}\) (width, height, and depth) is shared.  
In order for the CNN model to learn highly non-linear features, different activation functions have been incorporated into its architecture, with the typical functions being sigmoid, tanh, and ReLU. Subsequently, a pooling layer placed between two convolutional layers reduces the resolution of the feature maps. The pooling operation denoted as \(\text{pool}(\cdot)\) applied to each feature map \(\mathbf{h}^{l}_{i,\cdot,k}\) is defined as follows:  
\[t^{l}_{i,j,k}=\text{pool}(h^{l}_{m,n,k}),\quad\forall\ (m,n)\in\mathcal{R}_{ij} \tag{8}\]  
where \(\mathcal{R}_{ij}\) is a local region centered on location \((i,j)\). The most common pooling operations are min pooling, max pooling [59], and average pooling [60]. Filters in the first convolutional layer detect low-level features of the inputs, such as edges and curves, while filters associated with the deeper layers are trained to encode more abstract features [61].  
After adding several convolutional and pooling layers, one or more fully connected layers drive the final model decision through high-level feature learning. For instance, the feature extraction from the final convolutional layer can be coupled with the MLP approach. This class of models commonly employs the Softmax function to solve classification tasks [62]. The optimal parameter estimate can be obtained by minimizing the loss function defined in Equation (1). On the other hand, a global problem in deep learning is overfitting, tackled in the literature through regularization techniques. The typical regularizers are the \(\ell_{p}\)-norm, Dropout, and DropConnect. Note that, for \(p\geq 1\), the \(\ell_{p}\)-norm regularization is convex, while, for \(p<1\), the norm defines a non-convex regularization. The two special cases are \(p=1\) (\(\ell_{1}\)-norm) and \(p=2\) (\(\ell_{2}\)-norm), known as Tikhonov and LASSO regularization, respectively. The \(\ell_{2}\)-norm reduces the negative impact of noisy inputs, while the \(\ell_{1}\)-norm exploits the sparsity effect of the weights.