# Anomaly Detection of Wind Turbines Based on Deep Small-World Neural Network
## 2 Deep Small-World Neural Network (DSWNN)
### Pre-Training of the DSWNN Prototype  
As mentioned above, the DSWNN prototype is a deep auto-encoder network with multiple RBMs. An RBM is a specific energy-based stochastic model [24] with one visible layer and one hidden layer, in which the visible neurons \(v\) = (\(v_{1}\), \(v_{2}\),..., \(v_{m}\)) are connected fully to the hidden neurons \(h\) = (\(h_{1}\), \(h_{2}\),..., \(h_{n}\)). Figure 2 shows the mechanisms of an RBM.  
In the RBM, the energy of the joint configuration units \(E(v,h)\) is shown as Equation (1), and the joint probability \(P(v,h)\) between units based on the energy model is described as Equation (2).  
\[E(v,h)=-\sum_{i=1}^{n}\sum_{j=1}^{m}w_{ij}v_{i}h_{j}-\sum_{i=1}^{n}a_{i}v_{i}- \sum_{j=1}^{m}b_{j}h_{j} \tag{1}\]  
\[P(v,h)=\frac{e^{-E(v,h)}}{\sum\sum e^{-E(v,h)}} \tag{2}\]  
where \(w_{ij}\) is the symmetric weight between visible neurons \(i\) and hidden neurons \(j\), \(v_{i}\), \(h_{j}\) are the binary states and \(a_{i}\), \(b_{j}\) are their biases, respectively. The unbiased samples of \(v_{i}\) and \(h_{j}\) give a hidden vector \(h\) and a visible vector \(v\) can be calculated as follows:  
\[P(v_{i}=1|h)=f\left(\sum_{j=1}^{m}w_{ij}^{T}h_{j}+a_{i}\right) \tag{3}\]  
\[P(h_{i}=1|v)=f\left(\sum_{i=1}^{n}w_{ij}^{T}v_{i}+b_{j}\right) \tag{4}\]  
where \(f(\bullet)\) is an activation function that can be taken as a logistic Sigmoid function or \(tanh\) function, which are defined as Equation (5).  
\[f(\cdot)=\left\{\begin{array}{c}\text{Sigmoid}(x)=\frac{1}{1+e^{-x}}\\ tanh(x)=\frac{1-e^{-x}}{1+e^{-x}}\end{array}\right.. \tag{5}\]  
The essence of the activation function is to retain the characteristics of the activated neuron and map it out. The derivative of the log-likelihood with respect to weight \(w_{ij}\) is shown in Equation (6).  
\[\frac{\partial\ln P(v)}{\partial w_{ij}}=\left\langle v_{i}h_{j}\right\rangle_{ \text{data}}-\left\langle v_{i}h_{j}\right\rangle_{\text{model}} \tag{6}\]  
where, \(\left\langle v_{i}h_{j}\right\rangle_{\text{data}}\) represents the expectation with respect to the data distribution, and \(\left\langle v_{i}h_{j}\right\rangle_{\text{model}}\) represents the expectation with respect to the model distribution.  
However, getting an accurate description of \(\left\langle v_{i}h_{j}\right\rangle_{\text{model}}\) is computationally intractable, so [PERSON] proposed a Contrastive Divergence (CD) algorithm to crudely approximate the gradient [25]. First, the CD algorithm uses the input data to initialize the visible layer, and then calculates the hidden layer based on conditional distribution rules. Second, the visible layer is also calculated  
Figure 2: Structure of a Restricted Boltzmann Machine (RBM).  
according to the hidden layer. Relying on the repeated calculation, a reconstruction of the input data is obtained.  
\[\Delta w_{ij}=\eta\Big{(}\big{\langle}v_{i}h_{j}\big{\rangle}_{\text{data}}- \big{\langle}v_{i}h_{j}\big{\rangle}_{\text{recon}}\Big{)} \tag{7}\]  
where, \(\eta\) is the learning rate, \(\langle v_{i}h_{j}\rangle_{\text{recon}}\) is the expectation of reconstruction states, which is calculated by Gibbs sampler based on the initialized input data.  
The DSWNN prototype needs to pre-trained first, which is regarded as the process of training multiple RBMs layer by layer. The output of each RBM is considered as a new input of another RBM with a higher level to achieve the transmission of learning results. Once an RBM is trained, the weights between two-layer neurons are determined and locked. This procedure is conducted in an unsupervised environment, and the whole network weights will be obtained after the layer by layer pre-training. These weights will be recorded and used as the prior values for subsequent supervised training (fine-tuning).