## 4 Proposed Method
### Fault Prediction Method
#### 4.3.3 Attention-MLGRU Algorithm Flow  
The attention-based multi-layer GRU neural network is a recursive neural network structure that combines attention mechanisms for processing sequence data. By introducing attention mechanisms, it can automatically learn and focus on key information in the input sequence. 1. Forward propagation:  
It is assumed that there is a multi-layer GRU network with an L-layer, and the attention mechanism is introduced. The calculation process of each GRU unit can be expressed as:  
The reset gate of the GRU unit on level l:  
\[\mathrm{r}_{\mathrm{t}}^{(1)}=\sigma\Big{(}W_{\mathrm{r}}^{(1)}x_{\mathrm{t}} +U_{\mathrm{r}}^{(1)}h_{\mathrm{t-1}}^{(1)}+V_{r}^{(I)}c_{t}+b_{\mathrm{r}}^{ (1)}\Big{)} \tag{18}\]  
The update gate of the GRU unit on level l:  
\[\mathrm{z}_{\mathrm{t}}^{(1)}=\sigma\Big{(}W_{\mathrm{z}}^{(1)}x_{\mathrm{t}} +U_{\mathrm{z}}^{(1)}h_{\mathrm{t-1}}^{(1)}+V_{\mathrm{z}}^{(I)}c_{t}+b_{ \mathrm{z}}^{(1)}\Big{)} \tag{19}\]  
The candidate hidden state of the GRU unit on level l:  
\[\widetilde{h}_{\mathrm{t}}^{\mathrm{(l)}}=\tanh\Big{(}W_{\mathrm{h}}^{(1)}x_ {\mathrm{t}}+U_{\mathrm{h}}^{(1)}\Big{(}r_{t}^{(I)}\bigodot h_{t-1}^{(I)} \Big{)}+V_{\mathrm{z}}^{(I)}c_{t}+b_{\mathrm{h}}^{(1)}\Big{)} \tag{20}\]  
where \(x_{t}\) is the input vector, \(h_{t-1}^{(1)}\) is the hidden state of layer l at the previous moment, \(\mathrm{r}_{\mathrm{t}}^{(1)}\) is the reset gate, \(\mathrm{z}_{\mathrm{t}}^{(1)}\) is the update gate, and \(\widetilde{h}_{\mathrm{t}}^{(1)}\) is the candidate hidden state. Calculation of attention vector:  
\[e_{t}^{(l)}=tanh\Big{(}U_{a}^{(l)}h_{t-1}^{(l)}+V_{a}^{(l)}c_{t}+b_{a}^{(l)} \Big{)} \tag{21}\]  
\[\alpha_{t}^{(l)}=\frac{exp\Big{(}e_{t}^{(l)}\Big{)}}{\sum_{j=1}^{T}exp\Big{(} e_{j}^{(l)}\Big{)}} \tag{22}\]  
\[c_{t}=\sum_{j=1}^{T}\alpha_{t}^{(l)}\widetilde{h}_{j}^{(l)} \tag{23}\]  
where \(e_{t}^{(l)}\) is the intermediate result of the attention vector, \(\alpha_{t}^{(l)}\) is the attention weight, \(c_{t}\) is the attention-weighted context vector, and \(\widetilde{h}_{j}^{(l)}\) is the candidate always hidden state of the l-level GRU unit. Figure 15: **(D1)** GRU structure, **(D2)** MLGRU structure, **(D3)** Bi-GRU structure with attention mechanisms. Hidden state update:  
\[h_{\text{t}}^{\text{(I)}}=\left(1-z_{t}^{(l)}\right)\bigodot h_{\text{t-1}}^{ \text{(I)}}+z_{t}^{(l)}\bigodot\overset{\sim}{h_{t}}^{(l)} \tag{24}\]  
where \(h_{\text{t}}^{\text{(I)}}\) is the hidden state of layer I at the current moment. The forward propagation process of the entire multi-layer GRU neural network can be expressed as:  
\[h_{\text{t}}^{\text{(I)}}=\text{GRU}^{\text{(I)}}\left(x_{t},h_{t-1}^{(l)},c_{ t}\right) \tag{25}\]  
where \(\text{GRU}^{\text{(I)}}\) represents the GRU unit of layer I. 2. Calculate the loss function:  
Use an appropriate loss function, such as the mean square error loss, to calculate the error between the predicted value and the true label. 3. Backpropagation:  
According to the loss function, calculate the gradient of the loss relative to the parameter. Through the backpropagation algorithm, it is possible to calculate the gradient of the loss function to the network parameters and use the optimization algorithm (such as gradient descent) to update the parameters to train and optimize the network. 4.