# Predictive Maintenance Based on Identity Resolution and Transformers in IIoT
## 4 Proposed Framework for Predictive Maintenance
### State Awareness Model
These weights represent the association between the signal sampling points. These weights are then concatenated and multiplied with another matrix of weights \(W^{s}\) to obtain a representation matrix. This is shown in Equation (2). \[ \begin{split}&\text{Multi-head}(Q,K,V)=\\ &\text{Concat}(\text{Sdp}^{1},\text{Sdp}^{2},...,\text{Sdp}^{h}) \cdot W^{g}\end{split}  \tag{2}\]  
where \(W^{g}\in\mathbb{R}^{h\times d_{seq}\times d_{seq}}\). The authors set \(h\) to 6 in the proposed framework, which means that there are six parallel self-attention modules are used. Also, \(d_{b}=d_{c}=d_{seq}/6\). **(3) Feed Forward**  
In addition to the multi-headed attention layer, the transformer model contains a fully connected feed-forward network that operates uniformly on each subsegment. This feedforward network comprises a ReLU activation function and two linear transformations to introduce nonlinearity, as shown in Equation (3). \[ \begin{split}&\text{FF}(O_{mhd})=\\ &\text{max}(0,O_{mhd}\cdot W_{ff1}+bs_{ff1})\cdot W_{ff2}+bs_{ff2 }\end{split}  \tag{3}\]  
where \(O_{mhd}\) indicates the multi-head attention layer output. The \(W_{ff1}\), \(W_{ff2}\), \(bs_{ff1}\), and \(bs_{ff2}\) denote the weights and biases of the feed-forward layers, respectively. **(4) Residual Connection and Layer Normalization**  
As illustrated in Figure 2, the authors have used residual connections at every layer of the model [22]. Residual connection can be expressed mathematically as Equation (4). \[O_{lay}=\text{R}(I_{lay},W^{i})+I_{lay} \tag{4}\]  
where \(I_{lay}\) and \(O_{lay}\), respectively, denote the input and output of the currently designed layer. The weight matrix is denoted as \(W^{i}\). The residual representation to be trained is denoted as R. There are two main types of residual connections in this model, one of which is multi-head \((I_{lay})\) and the other is \(\text{FF}(I_{lay})\). Moreover, in addition to the residual connections, this paper also use layer normalization [23]. The operation of layer normalization is independent of the batch size setting. This is shown by the fact that the number of samples has no effect on the amount of data included in the layer normalization. The normalized statistics \(\mu\) and \(\sigma\) can be calculated according to Equations (5) and (6). \[\mu^{lq}=\frac{1}{Hq}\sum_{i=1}^{Hq}a_{i}^{lq} \tag{5}\]  
\[\sigma^{lq}=\sqrt{\frac{1}{Hq}\sum_{i=1}^{Hq}\left(a_{i}^{lq}-\mu^{lq}\right)^ {2}} \tag{6}\]  
where \(Hq\) denotes the quantity of hidden layer nodes. \(lq\) represents the quantity of layers of the front network structure. \(a^{lq}\) is the vector representation of the summed inputs to the neurons in that layer. The normalized value \(\bar{a}^{lq}\) can be calculated through \(\mu^{lq}\) and \(\sigma^{lq}\), as shown in Equation (7). \[\bar{a}^{lq}=\frac{a^{lq}-\mu^{lq}}{\sqrt{\left(\sigma^{lq}\right)^{2}+\epsilon}} \tag{7}\]  
where \(\epsilon\) is a decimal number that avoids division by 0. To ensure that the normalization does not corrupt the key information, the authors have designed the following parameters: gain \(g\) and bias \(b\). \(f\) denotes the set activation function. The final output \(O_{lan}\) of the layer normalization can be calculated according to Equation (8). \[O_{lan}^{lq}=f(g^{lq}\odot\bar{a}^{lq}+b^{lq}) \tag{8}\]  
**(5) Linear Layer and Cost Function**  
Following processing by the transformer model, the authors obtain a multi-dimensional representation matrix. Our goal, however, is to classify the device state, i.e., to divide the data into \(K_{e}\) classes. Therefore, the multi-dimensional representation matrix is transformed into a one-dimensional vector using a Flattening operation. The flattened vector is then multiplied by a learnable matrix to obtain an implicit representation \(O_{fla}\) of dimension \([d_{ff},K_{e}]\). This operation can be expressed mathematically as Equation (9). \[O_{fla}=\text{Flatten}(O_{tb})\cdot W_{l}+bs_{l} \tag{9}\]  
where the output vector of the transformer model is denoted as \(O_{tb}\).