# Fisher Discriminative Sparse Representation Based on DBN for Fault Diagnosis of Complex System
## 2 Deep Belief Networks (DBNs)
For any given set of state \((v,h)\) in an RBM model, the energy function is defined as  
\[E_{\theta}(v,h)=-\sum_{i=0}^{n_{v}}a_{i}v_{i}-\sum_{j=0}^{n_{h}}b_{j}h_{j}-\sum_ {i=0}^{n_{v}}\sum_{j=0}^{n_{h}}h_{j}w_{j,i}v_{i} \tag{1}\]  
where \(n_{v}\) and \(n_{h}\) represent the number of neurons in the visible and hidden layer, respectively; \(v=(v_{1},v_{2},\dots,v_{n})^{T}\) and \(h=(h_{1},h_{2},\dots,h_{n})^{T}\) are the state vector of neurons in the visible and hidden layer of the model, respectively; \(a=(a_{1},a_{2},\dots,a_{n})^{T}\) and \(b=(b_{1},b_{2},\dots,b_{n})^{T}\) are the bias of neurons; and \(W=\left(w_{i,j}\right)^{T}\) is the weight matrix of the units between the visible and hidden layers. Then \(\theta=(W,a,b)\) is the parameters set of the RBM model, which is the goal matrix of our training. Then the joint probability distribution function of state \((v,h)\) is calculated according to  
\[P_{\theta}(v,h)=\frac{1}{z_{\theta}}e^{-E_{\theta}(v,h)} \tag{2}\]  
where \(P(h_{k}=1|v\,)\) is a partition function. The probability distribution \(P_{\theta}(v)\) of the observed data \(v\) is the practical concerns, which is also the edge distribution of \(P_{\theta}(v,h)\), recognized as the likelihood function as follows:  
\[P_{\theta}(v)=\sum_{h}P_{\theta}(v,h)=\frac{1}{z_{\theta}}{\sum_{h}e^{-E_{ \theta}(v,h)}} \tag{3}\]  
Similarly:  
\[P_{\theta}(h)=\sum_{v}P_{\theta}(v,h)=\frac{1}{z_{\theta}}{\sum_{v}e^{-E_{ \theta}(v,h)}} \tag{4}\]  
According to the Bayes and sigmoid function, the neuron in the visible (hidden) layer is activated, with a value of 1, given a full state of neuros in the hidden (visible) layer by the probability \(P(h_{k}=1|v\,)\) (\(P(v_{k}=1|h\,)\)). \[P(h_{k}=1|v\,)=sigmoid(b_{k}+\sum_{i=1}^{n_{v}}w_{k,i}v_{i}) \tag{5}\]  
\[P(v_{k}=1|h\,)=sigmoid(a_{k}+\sum_{j=1}^{n_{h}}w_{j,k}h_{j}) \tag{6}\]  
Figure 2: Schematic representation of a deep belief network (DBN). The number of layers and nodes of each layer are an example; they can be defined as needed. The training goal was to determine parameter \(\theta\) to ensure the output of RBM was as similar as possible to the maximum of the likelihood \(\prod\limits_{i=1}^{n_{0}}P(v^{i})\). Since the log-likelihood gradient was initially equal to the components, the target can be translated to maximize the function \(\varsigma_{\theta,s}=\ln\prod\limits_{i=1}^{n_{0}}P(v^{i})\). Notably, the training process is more efficient when using a gradient descent based on the contrastive divergence (CD) algorithm to optimize this question. Then, the weighting matrix \(\theta\) can be updated according to Equation (7). The detailed proof can be seen in a previous study [40]. To enhance the learning rate and accuracy, [PERSON] et al. proposed a more practical algorithm named persistent contrastive divergence [41], which traced the states of persistent chains instead of searching the initial value of the Gibbs Markov chain at the training data vector. \[ \begin{array}{l}\frac{\partial\ln P(v)}{\partial w_{i}}\approx P(h_{i}=1 \Big{|}v^{(0)}\,)v^{(0)}_{j}-P(h_{i}=1\Big{|}v^{(k)}\,)v^{(k)}_{j}\\ \frac{\partial\ln P(v)}{\partial a_{i}}\approx v^{(0)}_{i}-v^{(k)}_{i}\\ \frac{\partial\ln P(v)}{\partial h_{i}}\approx P(h_{i}=1\Big{|}v^{(0)}\,)-P(h_ {i}=1\Big{|}v^{(k)}\,)\end{array}  \tag{7}\]  
DBNs are a cascade connection of stacked RBMs and are composed of multiple layers of stochastic and latent variables, which can be considered a special form of the Bayesian probabilistic generative model. The schematic diagram of the DBN model is shown in Figure 3. The hidden and visible layers are connected in order. [PERSON] proposed a greedy layer-wise learning algorithm to train the model from bottom to top and train only one layer at a time, with the parameters of the other layers being fixed. The outputs of layer \(l-1\) are the input of layer \(l\) used to train until the top level is reached. This is called the pretraining or feature-learning process, which determines the independent parameters in every RBM of the DBN.