# Predictive Maintenance Based on Identity Resolution and Transformers in IIoT
## 4 Proposed Framework for Predictive Maintenance
### State Awareness Model  
The proposed SAM, as shown in Figure 2, comprises three fundamental modules: Inputs, Encoder, and Outputs. The Inputs module primarily includes signal embedding. The Encoder module consists of Multi-head Attention, Feed Forward, Residual Connection, and Layer Normalization. The Outputs module includes a Linear Layer and a Cost Function. Each of these components will be detailed in the following sections. **(1) Signal Embedding in SAM**  
The signal embedding in SAM utilizes a technique that can be applied to various types of sensor sequence data, including vibration, current, and temperature. This method involves two stages: dataset segmentation and signal segment embedding. In this study, high-frequency vibration signals are used as an example to illustrate this technique. Firstly, the initial dataset is partitioned into subsets using equal and sliding segmentation techniques. The sensor signal sequence is constructed by equally dividing the original dataset, which contains a sufficient number of vibration signals from the signal collection. Secondly, each uniform-length segment is further divided into a specific number of equivalent subsegments. This number corresponds to the quantity used in multi-head attention. Subsequently, the subsegments are concatenated to produce an embedded matrix \(E_{n}^{m}\) with dimensions \(n\times m\). Here, \(n\) represents the number of heads, and \(m\) signifies the length of each subsegment. **(2) Multi-head Attention**  
Figure 2: The architecture of the state awareness model. As the data pass through the signal embedding, they are passed into the transformer model. The processed data will first pass through a self-attentive layer, also known as a one-scaled dot product within a layer of multi-headed attention. The self-attentive processing enables concentration on other sub-segments of the identical signal sequences, while representing different segments. As stated in reference [9], the attention function consists of mapping a set about pairs of key values and a query of an output, with every element consisting of a vector. For the weight matrices \(W_{Q}\), \(W_{K}\), and \(W_{V}\) in the model to be initialized, this initialization process is performed randomly. These weight matrices are then adjusted and optimized iteratively throughout the training. Next, each data vector passed into the model, i.e., the embedding of the sensor sequence (matrix \(E_{n}^{m}\)), is multiplied with \(W_{Q}\), \(W_{K}\), and \(W_{V}\). The corresponding query, key, and value vectors are generated. The input sequence is then traversed, scoring each subsegment against all others to determine the extent to which other subsegments should be focused on for the complete representation of a given segment. The output matrix for this layer is calculated using Equation (1). The main process is to first dot-product all the query vectors with the key vectors and then divide the values obtained by \(\sqrt{d_{k}}\). Finally, the Softmax function is used to process the result of the division to obtain the weights of attention. The \(\sqrt{d_{k}}\) is the scale factor set to guarantee a consistent gradient. \[ \begin{split}&\text{Sdp}^{i}=\text{Attention}(Q*W_{Q}^{i},K*W_{K}^ {i},V*W_{V}^{i})\\ &=\text{softmax}\left[\frac{(Q*W_{Q}^{i})\cdot(K*W_{K}^{i})}{ \sqrt{d_{k}}}\right]\cdot(V*W_{V}^{i}).\end{split}  \tag{1}\]  
In Equation (1), the \(W_{Q}^{i}\), \(W_{K}^{i}\), \(W_{V}^{i}\in\mathbb{R}^{d_{seq}\times d_{k}}\) are defined, where \(d_{seq}\) indicates sequence length, and \(d_{k}\) signifies the dimensionality of queries and keys. It is worth noting that the authors have previously defined three different matrices \(Q\), \(K\), and \(V\) to represent queries, keys, and values. However, in this paper, the contents of all three matrices are embeddings of signal data processed by the signal processing model. As previously described, the attention weights are obtained after the signal processing model has processed the signal data using the scaled dot product.