# Multi-Head Self-Attention-Based Fully Convolutional Network for RUL Prediction of Turbofan Engines
## 3 The Proposed Neural Network
### Training of MSA-FCN and RUL Prediction Procedure  
In this paper, MSA-FCN is trained using the Huber loss function [34]. This function is a piecewise loss function used for data-driven prediction tasks. By combining the characteristics of the Mean Absolute Error Loss (MAE Loss) and the Mean Squared Error Loss (MSE Loss), it can stably handle outliers in predicted and labeled values. It possesses a fast gradient descent speed and an appropriate gradient descent magnitude. The Huber loss function used is shown in Equations (17) and (18). \[L_{\Omega}=L_{\Omega}^{\prime}+\Big{(}\frac{\sigma}{2n}\Big{)}\overset{\Omega _{w}}{\sum}w^{2}, \tag{17}\]  
\[L_{\Omega}^{\prime}=\left\{ \begin{array}{l}\frac{1}{M}\overset{M}{\sum}_{i} \Big{[}0.5\Big{(}y_{i}^{tra}-y_{i}^{pre,tra}\Big{)}^{2}\Big{]},\qquad\Big{|}y_ {i}^{tra}-y_{i}^{pre,tra}\Big{|}\leq\delta\\ \frac{1}{M}\overset{M}{i}\Big{(}\delta\Big{|}y_{i}^{tra}-y_{i}^{pre,tra}\Big{|} -0.5\delta^{2}),\ \Big{|}y_{i}^{tra}-y_{i}^{pre,tra}\Big{|}>\delta\end{array}  \right., \tag{18}\]  
where the first and second terms in Equation (17) denote the Huber loss and L2 regularization, respectively. Meanwhile, \(\Omega\), \(\Omega_{w}\), and \(\sigma\) in Equation (17) denote the set of the trainable parameters, the set of the weights, and the penalty term, respectively. \(\delta\) is an adjustable hyperparameter in Equation (18). When \(\delta\) tends to zero, the loss function will approximate MAE Loss; when \(\delta\) tends to infinity, the loss function will approximate MSE Loss. \(M\) in Equation (18) is the number of training samples. Ultimately, the network parameters are updated, and the loss is minimized through the Adam optimizer and the back propagation algorithm. The training process of MSA-FCN is summarized by Algorithm 1. ```
Input: Training datasets \(\{A_{i},y_{i}^{tra}\}_{i}^{N_{tra}}\), hyperparameters of the network. Output: The trained MSA-FCN with the optimized trainable parameters. 1:Repeat:
2: Sample a batch of data \(A_{i}\) from Training datasets. 3: Use DMSAM_Data to process \(A_{i}\) through Equations (1)-(5) and \(Y=X+layernorm(Y)\). 4: Use MHFCN to process \(B_{i}\) through Equations (9)-(12). 5: Use DMSAM_Features to process \(C_{i}\) through Equations (1)-(5) and \(Y=X+layernorm(Y)\). 6: Use MLPM to process \(D_{i}\) for outputting the predicted \(y_{i}^{tra}\) through flatten operation and Equations (13)-(16). 7: Use the predicted \(y_{i}^{pre,tra}\) and the corresponding training label \(y_{i}^{tra}\) to calculate the loss function through Equations (17) and (18). 8: Update the trainable parameters and minimize the loss function through the Adam optimizer and the back propagation algorithm. 9:Until convergence of the Huber loss. ```  
**Algorithm 1**Training of MSA-FCN  
In this section, the RUL prediction procedure of the equipment is summarized. As shown in Figure 5, the procedure mainly contains three parts, i.e., the acquisition of the raw monitoring data (Part A), the offline training stage (Part B), and the online test stage (Part C). 1. [label=()]
2. Acquisition of the raw monitoring data. 1. The sensors are mounted on the corresponding components of the equipment. 2. The data reflecting the evolution of the components throughout their life cycles are recorded. 3. The raw monitoring data containing historical data and real-time data are obtained. 3. Offline training stage. Part B.1 Transformation between historical data and training datasets. 1. [label=()] 2. The sensor data that have a definite change trend along the time series are selected from the historical data. 3. The selected sensor data are normalized. 3. A sliding time window operation is used to generate training samples from the normalized data, and a piece-wise function is utilized to generate the corresponding RUL training labels. 4. The samples and the labels are integrated to form training datasets. Part B.2 Training process. 1. [label=()] 2. The proposed network of learning the degradation information are built to establish the proposed neural network. 3.