# Research on Fault Diagnosis of Rotating Parts Based on Transformer Deep Learning Model
## 2 Materials and Methods
### Feature Extraction Module
#### 2.4.1 Multi-Head Self-Attention Layer (MSL)  
Multi-Head Attention (MHA) is a core component of the Transformer model. It allows the model to focus on different attention heads when dealing with different aspects of information. By introducing multiple sets of query, key, and value parameters, multiple sets of (multi-head) attention weights are calculated in parallel and, finally, stitched together, so that the model can simultaneously focus on information in different subspaces. In this study, the detailed structure and principle are shown in Figure 3.  
The output \(X^{\prime}_{o1}\) from the previous layer is input into the feature extraction Module. First, it enters the multi-head self-attention layer and divides the last dimension of \(X^{\prime}_{o1}\) into \(h\) heads (the \(h\)-th head corresponds to \(X_{h}\)). Then, in each header, three values are calculated through three linear layers--query (\(Q_{h}\)), key (\(K_{h}\)), value (\(V_{h}\)). The calculation is as follows:  
\[Q_{h}=X_{h}W_{Q_{h}},K_{h}=X_{h}W_{K_{h}},V_{h}=X_{h}W_{V_{h}} \tag{5}\]  
where \(W_{Q_{h}}\), \(W_{K_{h}}\), and \(W_{V_{h}}\) are the weight matrices for \(Q_{h}\), \(K_{h}\), and \(V_{h}\), respectively, in the \(h\)-th attention head, with zero biases. The number of heads (\(h\)) in the hyperparameters is an important parameter that needs adjustment. In this study, \(h\) is chosen to be 8. The sizes for \(Q_{h}\), \(K_{h}\), and \(V_{h}\) are all \(\mathbb{R}^{M^{2}\times d_{h}}(d_{k}=\frac{D_{embed}}{h})\). Next, the similarity calculation is performed, and the \(Q_{h}\) and \(K_{h}\) dot products are multiplied to obtain the similarity score, which is divided by the scaling factor \(d\) for scaling calculation to make the network training more stable. Then, it is calculated by softmax to obtain the attention score Attention score A\({}_{h}\)The result is dotted with \(\mathrm{V}_{h}\) and input to the linear layer to obtain the output \(X_{\mathrm{oh}}\), which is calculated as follows:  
\[X_{\mathrm{oh}}=\mathrm{softmax}(\frac{\mathrm{Q}_{h}K_{h}^{\mathrm{T}}}{\sqrt{d _{k}}})\mathrm{V}_{h}W_{A} \tag{6}\]  
where \(d_{k}\) is the dimension of the query (or key), \(T\) represents the transpose operation, and \(W_{A}\) is the weight matrix for linear mapping. Then, concatenate the outputs of all heads and perform a linear transformation to map, obtaining the final output \(X_{\mathrm{o}H}\in\mathbb{R}^{M^{2}\times D_{\mathrm{embed}}}\) for the multi-head self-attention layer as follows:  
\[X_{\mathrm{o}H}=\mathit{Concat}(X_{\mathrm{o}1},X_{\mathrm{o}2},\dots,X_{ \mathrm{oh}})\cdot W_{H} \tag{7}\]  
where \(\mathit{Concat}\) denotes the concatenation of results along the last dimension, obtaining the final output of multi-head attention, and \(W_{H}\) is the weight matrix for linear mapping. Next, it enters the next layer of the network, namely layer normalization. The difference between layer normalization and batch normalization is that layer normalization normalizes all neurons in an intermediate layer. Layer normalization is defined by the following formula:  
Figure 3: Multi-head self-attention layer. \[\tilde{z}^{(i)}=\frac{z^{(i)}-\mu^{(i)}}{\sqrt{\sigma^{(i)}+\epsilon}}\odot\gamma+ \beta\Leftarrow LN_{\gamma,\beta}(z^{(i)}) \tag{8}\]  
where \(z^{(i)}\) represents the net input of the \(i\)-th layer nerve, \(\mu^{(i)}\) and \(\sigma^{(i)}\) are its mean and variance, \(\gamma\) and \(\beta\) represent the parameter vectors of scaling and translation, which are the same as the \(z^{(i)}\) dimension, and \(\epsilon\) is to prevent invalid calculation when the variance is 0.