# Multi-Head Self-Attention-Based Fully Convolutional Network for RUL Prediction of Turbofan Engines
## 5 Performance Comparison and Discussion
### Ablation Experiments for the Multi-Head Structure  
This section validates the impact of the multi-head structure on the RUL prediction performance of MSA-FCN by conducting relevant ablation experiments on the FD002 and FD004 datasets. To this end, seven comparative models were designed for RUL prediction, namely, SSD-SHF-SSF, SSD-SHF-MSF, SSD-MHF-SSF, MSD-SHF-SSF, MSD-MHF-SSF, MSD-SHF-MSF, and SSD-MHF-MSF. Herein, SSD, SHF, and SSF represent the Single-head Sensor Data Self-Attention Module, the Single-head Fully Convolutional Neural Module, and the Single-head Spatial Feature Self-Attention Module, respectively. MSD, MHF, and MSF stand for DMSAM_Data, MHFCN, and DMSAM_Features, respectively. The performance metric results for all models are summarized in Table 12.  
In terms of RMSE and Score, MSA-FCN achieves lower average values compared with SSD-SHF-SSF, which can be regarded as a completely single-head structured neural network. Specifically, on FD002, the average values of MSA-FCN are reduced by 8.28% and 58.60% for RMSE and Score, respectively, compared with SSD-SHF-SSF. On FD004, the reductions are even greater, at 15.30% and 76.38%, respectively. These results indicate that when processing datasets containing multiple failure modes, the multi-head structure \begin{table}
\begin{tabular}{c c c c c c c c c} \hline \hline \multirow{2}{*}{**Variant Model**} & \multicolumn{2}{c}{**RMSE (FD002)**} & \multicolumn{2}{c}{**Score (FD002)**} & \multicolumn{2}{c}{**RMSE (FD004)**} & \multicolumn{2}{c}{**Score (FD004)**} \\ \cline{2-9}  & **Mean** & **STD** & **Mean** & **STD** & **Mean** & **STD** & **Mean** & **STD** \\ \hline MSA-FCN & **17.165** & **0.213** & 1395.750 & **93.024** & 19.401 & **0.349** & **2184.660** & **158.393** \\ TFormer & 17.305 & 0.440 & 1400.253 & 93.802 & **19.282** & 0.412 & 2254.872 & 380.871 \\ Connection & 17.555 & 0.670 & 1448.130 & 160.395 & 20.431 & 0.562 & 2652.390 & 535.874 \\ Connect\_2 & 17.269 & 0.370 & **1375.160** & 104.910 & 19.502 & 0.596 & 2569.110 & 470.045 \\ W/O Connection & 17.733 & 0.710 & 1425.227 & 125.469 & 19.632 & 0.434 & 2555.606 & 645.774 \\ \hline \hline \end{tabular}
\end{table} 
Table 11: Impact of residual connections within DMSAM on the prediction performance of MSA-FCN. enables MSA-FCN to extract useful degradation representations from raw monitoring data, thereby effectively improving the accuracy of RUL prediction. Except for the Score average on FD002, the remaining averages achieved by MSA-FCN are the lowest among all comparison models. This result suggests that DMSAM_Data, MH-FCN, and DMSAM_Features, constructed using the multi-head structure, enable MSA-FCN to enrich the extracted representations, leading to more timely RUL predictions. Finally, the STD values obtained by MSA-FCN are the lowest among all the comparison models, indicating that the multi-head structure enhances the stability of its RUL prediction results. In summary, the multi-head structure allows MSA-FCN to focus on multiple aspects of raw monitoring data including different degradation trends, stages, and interactions among components. Therefore, MSA-FCN, based on the multi-head structure, can achieve more accurate, timely, and stable results in RUL prediction. In this section, this paper also investigates the influences of heads of the multi-head structure on RUL prediction performance on FD004. Tables 13-15 summarize the experimental results for the three modules (DMSAM_Data, MHFCN, and DMSAM_Features) using different numbers of heads, respectively. Figure 10a-c are also used to further illustrate the experimental results. As can be seen from the above tables and figures, to achieve the lowest mean values on both RMSE and Score, the optimal number of heads for each model is as follows: DMSAM_Data uses 6 heads, MHFCN uses 3 heads, and DMSAM_Features uses 24 heads.