# An Improved Deep Belief Network Prediction Model Based on Knowledge Transfer
## 2 Related Works
### Unsupervised Learning Based on RBM  
Unsupervised learning is a process of obtaining initial weights by training stacked RBMs layer by layer. The basic structure of RBM is shown in Figure 2. Given a set of states \(h~{}=~{}(h_{1},h_{2},\ldots,h_{m})\) and a set of parameters \(\theta~{}=~{}\Big{(}w_{ij},b_{i},c_{j}\Big{)}\), the energy function and joint probability distribution are defined as follows:  
\[E(v,h|\theta)~{}=~{}-\sum_{i~{}=~{}1}^{n}b_{i}v_{i}-\sum_{j~{}=~{}1}^{m}c_{j}h_{ j}-\sum_{i~{}=~{}1}^{n}\sum_{j~{}=~{}1}^{m}v_{i}w_{ij}h_{j} \tag{1}\]  
\[P(v,h|\theta)~{}=~{}\frac{\exp(-E(v,h|\theta))}{Z(\theta)} \tag{2}\]  
where \(Z(\theta)\) is the partition function, \(v\) is the visible layer unit vector, \(h\) is the hidden layer unit vector, and \(w_{ij}\) is the weight matrix connecting the visible layer and the hidden layer, \(b_{i}\) and \(c_{j}\) are the bias vectors of the visible layer unit and the hidden layer unit, respectively. Calculate the marginal probability distribution about \(v\) through the joint probability distribution:  
\[P(v|\theta)~{}=~{}\frac{\sum\limits_{h}\exp(-E(v,h|\theta))}{Z(\theta)} \tag{3}\]  
To update the weight parameters of RBM, the partial derivative with respect with respect to \(P(v|\theta)\) needs to be calculated. The parameter update rules are as follows:  
\[\Delta\theta~{}=~{}\frac{\partial\log_{p}(v,h)}{\partial\theta}~{}=~{}E_{data }\Big{(}v_{i}h_{j}\Big{)}-E_{\text{model}}\Big{(}v_{i}h_{j}\Big{)} \tag{4}\]  
\[\theta_{i+1}~{}=~{}\theta_{i}+\mu\Delta\theta \tag{5}\]  
where \(\mu\) is the learning rate of the model. Since the second part of formula (4) contains the high computational complexity of the partition function, the CD algorithm is usually used to approximate \(E_{\text{model}}\). The training process of the CD algorithm only needs K steps (usually K = 1) Gibbs sampling to get good results. The sampling process of K-CD algorithm is shown in Figure 3.  
Figure 2: RBM structure.  
In the sampling process, when the state of the visible layer unit is given, the activation probability of the \(j\)-th hidden layer unit is:  
\[P\big{(}h_{j}\ =\ 1\big{|}v,\theta\big{)}\ =\ o\Bigg{(}c_{j}+\sum_{i}v_{i}w_{i,j} \Bigg{)} \tag{6}\]  
where \(\sigma(x)\ =\ \frac{1}{1+e^{x}}\). Similarly, when the state of the hidden layer unit is given, the activation probability of the \(i\)-th visible layer unit is:  
\[P(v_{i}\ =\ 1\big{|}h,\theta)\ =\ o\Bigg{(}b_{i}+\sum_{j}w_{i,j}h_{j}\Bigg{)} \tag{7}\]  
Since the RBM unit state value is binary. Taking the hidden layer unit as an example, the state value of the hidden layer unit \(h_{j}\in\{0,1\}\) is determined by the random number \(r_{j}\in[0,1]\), and the value state of \(h_{j}\) is as follows:  
\[h_{j}\ =\ \Bigg{\{}\begin{array}{l}1,\ p\big{(}h_{j}\ =\ 1\big{|}v\big{)}>r_{j}\\ 0,\ otherwise\end{array} \tag{8}\]  
In the unsupervised learning process, the greedy layer-by-layer training algorithm is used to train each layer of RBM from low to high. The hidden layer of each RBM is mapped to a different feature space. The higher the hidden layer, the more abstract features are obtained. The last hidden layer and the output layer of the top layer constitute an associative memory layer, which can correlate the optimal parameters of the remaining layers. Through unsupervised learning, DBN can obtain prior knowledge. In this paper, the initial weight obtained by pre-training is taken as an important knowledge in the process of knowledge transfer, which is the premise to determine the optimal network structure of DBN.