# Predictive Maintenance Based on Identity Resolution and Transformers in IIoT
## 5 Evaluation
### Experiments on Sequence Prediction Model  
For the second experiment, the authors conduct relevant experiments to evaluate the perception capacity of the SPM. To comprehensively assess model performance, the varied performance metrics and the CWRU dataset are employed for both training and testing processes. The experimental results, presented in Table 4, demonstrate that optimal model performance is achieved when sequence length is set to 1024, as confirmed by a lower RMSE.  
Figure 5: Metrics of the state awareness model during training and testing. (**a**) Training accuracy and loss. (**b**) Accuracy, precision, recall, F1-score, and rest consumption in testing. (**c**) TPR, FPR, and AUC in testing.  
\begin{table}
\begin{tabular}{c c c c} \hline \hline
**Method** & **Epoch** & **Number of Species** & **Accuracy** \\ \hline biLSTM & & & 98.041\% \\ biGRU & 300 & 10 & 98.623\% \\ SAM & & & **99.062\%** \\ \hline \hline \end{tabular}
\end{table}
Table 3: The evaluation of the state awareness model compare with other methods.  
Figure 6: Comparison of methods on the CWRU dataset.  
Furthermore, the proposed model illustrates a good convergence effect for the model, as shown in Figure 6(a), which showcases the RMSE and MAE during the training process. In Figure 6(b), the authors observe a monotonic growth curve in the MAE, while the RMSE arrives at its minimal value when the length of the sequence is 1024. Taken together, the SPM attains superior predictive performance at sequence length set to 1024, as supported by the combination of both measured metrics.  
To further validate the efficacy of the SPM, the authors selected ten representative samples from the CWRU dataset, as described in Figure 4, across different categories. These samples were taken from the predicted sequences and compared to their corresponding original samples, as visualized in Figure 8. The blue line represents the real data, while the red line represents the predicted data. The close alignment between the predicted and original sequences strongly supports the predictive capability of the proposed sequence prediction model, demonstrating a robust fit between both sets of sequences.  
To summarize, the results confirm that the proposed sequence prediction model is fundamental in achieving real-time PdM and can produce future sensor sequences based on current sensor data. The ITPM framework showcases superior performance outcomes. However, this study has several limitations and practical challenges. The sample selection bias may exist as the experimental equipment and sensor signals were chosen from a specific industrial environment, potentially limiting the generalizability of results. Future research should consider a wider range of equipment types and operational environments to validate the universality of the framework. Measurement errors and data noise, despite mitigation efforts, are additional factors that may affect results, necessitating further optimization.  
\begin{table}
\begin{tabular}{c c c c c c} \hline \hline \multirow{2}{*}{**Sequence Length**} & \multirow{2}{*}{**Data Volume**} & \multicolumn{2}{c}{**Time Consumption (s)**} & \multicolumn{2}{c}{**Metrics (Testing)**} \\ \cline{3-6}  & & **Train** & **Test** & **RMSE** & **MAE** \\ \hline
256 & 18,880 & 9656.101 & 1.418 & 25.050 & **0.629** \\
512 & 16,100 & 10,699.200 & 1.416 & 17.986 & 1.088 \\
1024 & 16,080 & **3081.609** & **1.268** & **14.522** & 2.235 \\
2048 & 16,050 & 4527.055 & 1.382 & 21.917 & 4.637 \\
4096 & 15,980 & 6093.705 & 1.392 & 33.890 & 8.693 \\ \hline \hline \end{tabular}
\end{table}
Table 4: The evaluation results of the sequence prediction model.  
Figure 7: Metrics of the sequence prediction model during the training and testing.