## 2 Deep Learning  
Deep belief networks (DBNs) are regarded as the pioneering model of deep learning practices, as when first introduced [8; 9] they were the first deep model that upon successful training managed to outperform Support Vector Machines (SVMs) in a classification problem [10]. DBNs represent generative models of probabilistic nature which contain a population of latent, stochastic parameters. These parameters are used as characteristics detectors of higher order correlations into the training/testing datasets. The DBN is created when several Restricted Boltzmann Machines (RBMs) are stacked in a sequential manner [11]. This architecture is formed by the principle that the hidden layer of \(RBM_{i-1}\) becomes the seeable one for \(RBM_{i}\). Furthermore, note that in DBNs, all layers have directed connections except for the last two which have undirected ones [12]. Let us consider a representation of such a network composed by five layers of size \(L1\), \(L2\), \(L3\), \(L4\) and \(L5\), respectively, where a quadruple of RBMs construct a DBN. \(RBM_{1}\) consists of layers \(L1\) and \(L2\) while the remaining RBMs are determined correspondingly. Based on the previous description, \(L2\) is the hidden layer for \(RBM_{1}\) while it constitutes also the visual one for \(RBM_{2}\). As mentioned before, training of deep networks of such architecture was not possible until the training methodology proposed by [PERSON] [10] was published. This methodology consists of a two-step training procedure: in the first one, termed as pre-training, each RBM of the DBN is trained according to an unsupervised manner, while in the second one, labeled as fine-tuning, the whole DBN undergoes supervised training [10; 13]. Pre-training is performed by applying the contrastive divergence method in each RBM, sequentially. For \(RBM_{1}\), the visible layer refers to the actual input while the hidden layer, once trained,will act as the visual layer in the training of \(RBM_{2}\) as described previously. The weight coefficients (\(w_{ij}\)) of every RBM's links are updated as follows:  
\[ \begin{split}\frac{\partial\log p(v;\theta)}{\partial w_{ij}}& =\big{\langle}v_{i}h_{j}\big{\rangle}_{\text{input}}-\big{\langle}v_{i}h_{ j}\big{\rangle}_{\text{model}}\\ w_{ij}^{new}&=w_{ij}+e\Delta w_{ij}\end{split}  \tag{2}\]  
where \(p()\) denotes the state probability of the network; \(\theta\) represent the set of weight and bias coefficients; \(v_{i}\) and \(h_{j}\) denote the state of \(i\)th and \(j\)th unit of the visible (\(v\)) and hidden (\(h\)) layer, respectively; and \(e\) denotes the weight learning rate and defines the range of weight changes. Fine-tuning is performed once pre-training is finished; the latter is performed by means of the back propagation algorithm [14] combined with the conjugate gradient algorithm [15]. A similar architecture to DBNs are the Deep Boltzmann Machines (DBMs), introduced in 2009 [16], which also are probabilistic graphical models with multiple layers of stochastic nodes. DBNs and DBMs share many common features in terms of architecture and functionality, where there are no connections between nodes of the same layer and the adjacent ones are fully connected, i.e., all nodes of a specific layer are linked to all nodes of the next and previous layers. The main difference is the fact that all layers of a DBM are undirected. Apart from the first layer, also known as visible, all other layers of the network are hidden. A simple DBM with five layers can be seen in Figure 1. As energy-based models, an energy function is used for defining the joint probability distribution of the variables, similarly to DBNs [12]. The energy of a DBM like the one presented in Figure 1 can be calculated as follows:  
\[ \begin{split} E(v,h^{1},h^{2},h^{3},h^{4},h^{5};\theta)& =-v^{T}W^{1}h^{1}\\ &-h^{1}W^{2}h^{2}-h^{2}W^{3}h^{3}\\ &-h^{3}W^{4}h^{4}-h^{4}W^{5}h^{5}\end{split}