# Development of Fingerprint Identification Based on Device Flow in Industrial Control System
## 4 Simulation Experiments
### Experimental Results and Performance Analysis
#### 4.3.2 Performance Analysis of Device Traffic Fingerprint Model
For SVM and random forest models, the features extracted from different industrial control protocols are not exactly the same, and the recognition results will also be different. More features that are helpful for recognition can be extracted from the s7 comm protocol, so the recognition effect is optimal. Figure 13 describes the relationship between the recognition accuracy of each model and the number of device fingerprints in different industrial control protocols. Merely calculating the accuracy rate is not enough to evaluate the recognition ability of a model. The accuracy rate can well reflect the model's ability to distinguish negative samples. The higher the accuracy rate, the stronger the model's ability to distinguish negative samples. The highest accuracy rate is the device traffic fingerprint model, which is between 0.9725  
Figure 12: Accuracy curves of the four models. and 0.995. The accuracy of the LSTM model is slightly lower than that of the traffic fingerprint model, between 0.95 and 0.98. The accuracy rates of the random forest and SVM models are relatively low, and the accuracy rates drop significantly when the number of device fingerprints is 7. The reason is the same as the drop in accuracy rates. The results show that the device traffic fingerprint model has a strong ability to distinguish negative samples. Figure 14 depicts the relationship between the recognition recall of each model and the number of devices for different industrial control protocols. Recall is a measure of coverage that measures how well a classifier can identify positive examples. The recall rate of the device fingerprint model is about 0.78-0.89, and the recall rate of the random forest model and the recall rate of the SVM model are about 0.71-0.82, but the problem of equipment confusion fluctuates greatly. The LSTM model still performs well in the recall rate, which is about 0.02 different from the fingerprint model. The results show that the device fingerprint model has excellent performance in terms of recall rate and strong ability to identify positive samples. Figure 15 depicts the ROC plots for the four models under the four devices. The abscissa is fpr, which indicates the proportion of samples that are predicted to be positive, but actually negative to all negative samples. The ordinate is fpr, which indicates the proportion of samples that are predicted to be positive and actually positive to all positive samples. The ROC curve combines the true positive rate and the false positive rate, which can accurately reflect the impact of any threshold on the generalization performance of the model, and help to choose the best threshold. The point on the ROC curve closest to the upper left corner is the best threshold with the least classification error, and the total number of false positives and false negatives is the least. The larger the area under the ROC curve, the better the performance of the model. From Figure 15a, it can be seen that the four ROC curves of the device fingerprint model are all close to the upper left corner, the highest area value is 1.00, the lowest is 0.94, and the comprehensive area value is 0.98. The performance of the random forest model and the SVM model is slightly worse, with the comprehensive area values of 0.94 and 0.92, respectively. The LSTM model still shows good  
Figure 13: Precision curves of the four models. performance, the highest area value is 0.97, the lowest is 0.96, and the comprehensive area value is 0.97, which is very close to the recognition effect of the device fingerprint model. In the above experiments, the device fingerprint model shows excellent performance in terms of accuracy, precision, recall and ROC curve. The performance of random forest is slightly stronger than that of the SVM model, but it is inferior to the device fingerprint model in various indicators. The performance of the LSTM model is very close to the device fingerprint model, with a maximum difference of about 0.02 for each index.