# An Attention-Based Method for Remaining Useful Life Prediction of Rotating Machinery
## 2 Attention-Based RUL Prediction of Rotating Machinery
### Attention-Based GRU for RUL Prediction
#### 2.2.2 Attention-GRU Model for RUL Prediction  
The RUL prediction attention-GRU model consists of two modules: a GRU module and an attention module. The GRU module learns and analyzes the sequence trends and intrinsic relationships of data; it can independently determine whether to retain or discard a feature and is mainly used for predictive calculations. The attention module is used for similarity weights (Figure 5). The attention mechanism gives a larger weight to the part of attention than to the less relevant parts to obtain more effective information. Therefore, adding an attention layer to the GRU model can increase the contribution of important features to the prediction. (1) As shown in Figure 5, in contrast to traditional RNNs, the GRU module has only a reset gate (\(R_{g}\)) and an update gate (\(U_{g}\)); \(X_{t}\) is the current input feature; the final output state of the module is the weighted sum of the previous state (\(H_{t-1}\)) and the candidate state (\(\widetilde{H_{t}}\)). The reset gate \(R_{g}\) processes the previous state (\(H_{t-1}\)); it receives the current input feature and the previous state and performs a linear calculation. Then, the sigmoid function is employed for normalization. The value range of the reset gate is between 0 and 1; a larger value indicates a higher proportion of the previous state in the candidate state. The value  
Figure 5: Structure of the attention–GRU model. of the reset gate determines the importance degree of the previous state and the current input feature in the candidate state (\(\widetilde{H_{t}}\)). Let \(\sigma\) be the sigmoid function; \(\tan\mathrm{h}\) be the hyperbolic tangent function; \(W_{r}\), \(W_{u}\), and \(W_{h}\) be the network weights, where \(W_{r}\) is the network weight of the reset gate, \(W_{u}\) is the weight of the update gate, and \(W_{h}\) is the network weight of candidate output (\(\widetilde{H_{t}}\)); \(W_{xr}\) and \(W_{hr}\) be the weight parameters of \(W_{r}\); \(W_{xu}\) and \(W_{ht}\) be the weight parameters of \(W_{hi}\), and \(W_{xh}\) and \(W_{hh}\) be the weight parameters of \(W_{hh}\), as shown in Equation (5):  
\[ \begin{array}{l}W_{r}=W_{xr}+W_{hr}\\ W_{u}=W_{xu}+W_{hu}\\ W_{h}=W_{xh}+W_{hh}\end{array}  \tag{5}\]  
Let \(b_{r}\) and \(b_{h}\) be the bias; then, the reset gate and the candidate state can be calculated by Equations (6) and (7), respectively:  
\[R_{g}=\sigma(X_{t}W_{xr}+H_{t-1}W_{hr}+b_{r}) \tag{6}\]  
\[\widetilde{H_{t}}=\tan\mathrm{h}\Big{(}X_{t}W_{xh}+\Big{(}R_{g}^{T}H_{t-1} \Big{)}W_{hh}+b_{h}\Big{)} \tag{7}\]  
The update gate (\(U_{g}\)) assigns weights to the input features and the previous state to update the output state (\(H_{t}\)). The calculation of the update gate resembles that of the reset gate. The update gate also determines the candidate output (\(\widetilde{H_{t}}\)) and the impact of the previous output on the final output (\(H_{t}\)). The calculation of the update gate and the final estimation state (\(H_{t}\)) is shown in Equations (8) and (9), respectively:  
\[U_{g}=\sigma(X_{t}W_{xu}+H_{t-1}W_{hu}+b_{h}) \tag{8}\]  
\[H_{t}=U_{g}^{T}H_{t-1}+\big{(}1-U_{g}\big{)}^{T}\widetilde{H_{t}} \tag{9}\]  
(2) In the attention module, the current input feature (\(X_{t}\)) is considered the inquiry vector (\(Query\)), the \(k\) groups of previous states (\(H=\{H_{t-k},H_{t-k+1},\ldots,H_{t-1}\}\)) are considered the value vector (\(Value\)), and \(Value\) per se is considered the key vector (\(Key\)) for the similarity calculation. The attention calculation involves three parts: first, the dot product of \(Query\) (\(X_{t}\)) and \(Key\) (\(H\)) is calculated to obtain the weight coefficient; then, Softmax is employed to normalize the weight coefficients; finally, the adjusted weight coefficients are used to perform weighted summation of \(Value(H)\).