## 3 Background
### Deep Belief Networks  
Deep belief networks (DBNs) are multi-layer probabilistic generative models that can learn to extract a deep hierarchical representation of the training data. DBNs are composed of several layers of restricted Boltzmann machines (RBMs) and a classifier added on the top layer. DBNs can be quickly trained by training several RBMs using the greedy layer-wise unsupervised training strategy [16; 40]. After network pretraining, the network parameters using supervised learning are fine-tuned to achieve better classification results. RBMs are an energy model in which the visible-hidden layers have full connections and the visible-visible and hidden-hidden layers have no connections. RBMs have been widely used in classification, dimension reduction, feature extraction, topic modeling and collaborative filtering. An RBM has visible and Bernoulli random-valued hidden units, and maps the sample data from an input space of dimension \(m\) to the feature space of dimension \(n\), where \(n<m\), as shown in Figure 2. An RBM is an energy-based generation model that contains a layer of visible nodes \((v_{1},v_{2},\cdots,v_{i},\cdots,v_{m})\) representing the data and a layer of hidden nodes \((h_{1},h_{2},\cdots,h_{j},\cdots,h_{n})\) learning to represent features, with each \(v_{i}\in\{0,1\}\) and \(h_{j}\in\{0,1\}\). We define that the biases of the  
Figure 1: The nearest neighbor distance of the test sample in each cluster. visible nodes are \((b_{1},b_{2},\cdots,b_{i},\cdots,b_{m})\), and the biases of the hidden nodes are \((c_{1},c_{2},\cdots,c_{j},\cdots,c_{n})\). The energy function \(E(v,h)\) of the joint configuration \(\{v,h\}\) is defined as follows [41; 42]:  
\[E(v,h)=-\sum_{i=1}^{m}b_{i}v_{i}-\sum_{j=1}^{n}c_{j}h_{j}-\sum_{j=1}^{n}\sum_{i =1}^{m}v_{i}w_{ij}h_{j}. \tag{15}\]  
According to the energy function \(E(v,h)\), the joint probability distribution for visible and hidden vectors can be defined as follows:  
\[P(v,h)=\frac{1}{Z}\exp(-E(v,h)), \tag{16}\]  
where \(Z\) is the partition function, which is the sum over all possible pairs of visible and hidden vectors. \[Z=\sum_{v}\sum_{h}\exp(-E(v,h)). \tag{17}\]  
The probability assigned to a visible vector \(v\) is given by summing over all possible binary hidden vector \(h\), as follows:  
\[P(v)=\sum_{h}P(v,h)=\frac{1}{Z}\sum_{h}\exp(-E(v,h)). \tag{18}\]  
Since there are no direct connections between the same layers in an RBM, these hidden units are independent given the visible units, and vice versa. Therefore, given the visible vector \(v\), the conditional probability of the hidden vector \(h\) is given by:  
\[P(h|v)=\prod_{j=1}^{n}P(h_{j}|v). \tag{19}\]  
Similarly, given the hidden vector \(h\), the conditional probability of the visible vector \(v\) is given by:  
\[P(v|h)=\prod_{i=1}^{m}P(v_{i}|h). \tag{20}\]  
Given a visible vector \(v\), the activation state of each hidden unit is conditionally independent. At this point, \(h_{j}\in\{0,1\}\), and the activation probability of the \(j\)th hidden unit is described as follows:  
\[P(h_{j}=1|v)=Sign(\sum_{i=1}^{m}w_{ij}v_{i}+c_{j}), \tag{21}\]  
where \(Sign(x)=\frac{1}{1+e^{-x}}\) is the logistic sigmoid function. Figure 2: The graphical representation of an RBM with \(m\) visible nodes and \(n\) hidden nodes. Accordingly, the activation probability of each visible unit is also conditionally independent when given a hidden vector \(h\):  
Figure 1:0  
An RBM is trained to minimize the energy in Equation (15) by finding the values of the network parameters \(\theta=(W,b,c)\). If the RBM has been trained, the energy of the network is minimized, and the probability in Equation (18) is maximized. To maximize the log-likelihood of \(P(v)\), its gradient with respect to the network parameters \(\theta\) can be calculated as follows [36; 43]:  
Figure 1:1  
where \(\mathbb{E}\) denotes the expectation operator. In Equation (23), the expectation on the left hand side can be calculated exactly, but the expectation on the right hand side is hard to calculate.