# Multi-Head Self-Attention-Based Fully Convolutional Network for RUL Prediction of Turbofan Engines
## 3 The Proposed Neural Network
### Architecture of MSA-FCN
#### 3.2.1 Multi-Head Fully Convolutional Network (MHFCN)  
Within a piece of equipment, the degradation of one component not only causes the degradation of other components but also is affected by the degradation of other components. This degradation correlation among components implies that spatial characteristics are embedded in the corresponding sensor data. Therefore, MHFCN is constructed by integrating a fully convolutional network with a multi-head structure to take into account relevant spatial characteristics. As shown in Figure 3, MHFCN utilizes multiple fully convolutional networks to perform convolutional operations on the input in both the sensor and spatial dimensions, extracting the degradation representations with spatial characteristics from the sensor weighted data. Simultaneously, by organizing these fully convolutional networks in parallel through the multi-head structure, MHFCN can effectively enrich the representations it extracts, providing support for subsequent RUL prediction.  
The forward propagation of MHFCN is described as follows:  
1. First, in each head, the mappings of the first layer are obtained by processing the weighted sensor data \(B_{i}\) along the sensor dimension. Each head of MHFCN (each fully convolutional network) uses convolutional kernels to perform convolution operations on multiple different sensor data points at each time point within the weighted data \(B_{i}\) along the sensor dimension and traverses \(B_{i}\) through the operation of sliding convolutional kernels, thereby obtaining the first layer mapping of the corresponding fully convolutional network. The steps above are shown in Equations (9) and (10). \[y_{i}=W_{i}*\text{LeakyReLU}(B_{i})+b_{i},padding=``same\",stride=1,\] (9) \[Y=[y_{1},\dots,y_{i},\dots,y_{N_{filters}}],\] (10) where \(W_{i}\), \(b_{i}\), \(y_{i}\), LeakyReLU, \(*\), \(N_{filters}\), and \(Y\) represent the weight of the \(i\)th filter, the bias corresponding to \(W_{i}\), the mapping of the \(i\)th filter, the activation function, the convolution operation of the \(i\)th filter, the number of the filters, and the mapping of Conv1D, respectively. It is worth noting that the aforementioned steps adopt the pre-activation function strategy [32], which helps capture the nonlinear relationships among different sensor data points and can enhance the modeling capabilities of MSA-FCN.
2. MHFCN performs three iterative processings of the data along the spatial dimension using Equations (9) and (10), thereby obtaining deep mappings on each head of MHFCN.
3. Each head of MHFCN combines the weighted data \(B_{i}\) with the aforementioned mappings through residual connection [32] to extract the corresponding spatially characteristic degradation representations. Residual connection can reduce the training burden of neural networks and enhance the stability of MSA-FCN predictions for RUL. The calculation is shown in Equation (11). \[\widetilde{C}_{h}=\widetilde{C}_{h,L}+B_{i},\] (11) where \(\widetilde{C}_{h,L}\) and \(\widetilde{C}_{h}\) represent the mappings learned by the \(h\)th head and the degradation representations with spatial features learned by the \(h\)th head, respectively. (4) MHFCN uses a concatenation layer to aggregate the representations extracted from all its heads, thereby obtaining the final degradation representations with spatial characteristics \(C_{i}\). The calculation is shown in Equation (12). \[C_{i}=concatenate\left(\widetilde{C}_{1},\ldots,\widetilde{C}_{h},\ldots, \widetilde{C}_{N_{h,fcn}}\right),\] (12) where \(N_{h,fcn}\) represents number of the heads of MHFCN.