# Multi-Head Self-Attention-Based Fully Convolutional Network for RUL Prediction of Turbofan Engines
## 3 The Proposed Neural Network
### Architecture of MSA-FCN
#### 3.2.2 Dual Multi-Head Self-Attention Module (DMSAM)  
Importance differences in degradation information exist in both the sensor data and representations extracted by the deep neural network. However, some existing models for the RUL prediction of turbofan engines only consider the differences within the sensor data or within the extracted representations, which may result in the loss of some degradation information that can contribute significantly to the corresponding RUL prediction. To  
Figure 3: Diagram of MHFCN.  
alleviate this challenge, DMSAM is built based on Transformer's multi-head self-attention module. It aims to enhance the key parts of the data and representations that significantly contribute to RUL prediction by learning and allocating relevant attention weights, thus providing support for MSA-FCN's RUL prediction. As shown in Figure 4, DMSAM consists of two modules, DMSAM_Data and DMSAM_Features, which process various sensor data points and representations extracted by MHFCN, respectively. Specifically, DMSAM_Data and DMSAM_Features process the input data and representations along the sensor and spatial dimensions, respectively, to learn the relationships between different data points and representations, dynamically compute relevant attention weights, and allocate those weights to the corresponding parts within them. Through the above processes, DMSAM_Data and DMSAM_Features can effectively generate weighted sensor data and weighted degradation representations with spatial characteristics, respectively. They share the same network structure, but the former uses more heads than the latter to adapt to the increased number of representations after MHFCN processing. The residual connection of both modules is performed after layer normalization, which is different from the residual connection in Transformer. This can be expressed as \(Y=X+laynorm(\widetilde{Y})\).  
DMSAM_Features and DMSAM_Data are identical in terms of the mechanism of data processing. Therefore, only the forward propagation of DMSAM_Features is described below.  
1. DMSAM_Features has three computational steps in each of its heads (each spatial feature self-attention module). 1. Each head performs a linear projection of the degradation representations \(C_{i}\) along the spatial dimension. This step is shown in Equation (1). 2. Each head applies a scaled dot-product self-attention mechanism in the spatial dimension to process the obtained query and key matrices, thereby learning the relationships among various representation parts. Based on these relationships, attention weights reflecting the importance of different representation parts are dynamically calculated and obtained. This step is shown in Equation (2). 3. Each head assigns the obtained attention weights to the corresponding representation parts. This step is shown in Equation (3). 2. DMSAM_Features utilizes a concatenation layer to aggregate the weighted degradation representations extracted from all its heads along the spatial dimension. This step is shown in Equation (4). 3. DMSAM_Features performs a linear projection on the aggregated output to ensure it maintains the same dimensions as \(C_{i}\) along the spatial dimension. This step is shown in Equation (5).  
Figure 4: Diagram of DMSAM.  
* [(4)] DMSAM_Features combines the reshaped outputs and \(C_{i}\) using a residual connection \(Y=X+laynorm(\overset{\sim}{Y})\) to extract the final weighted degradation representations with spatial characteristics \(D_{i}\).