# Enhancing Security in Smart Robot Digital Twins Through Intrusion Detection Systems
## 5 Experimental Results
### Analysis of the CICIDS2017 Dataset  
We begin with the results of binary classification (abnormal behavior detection), specifically focusing on the performance of SVM on the validation set. The results obtained by SVM on this set are as follows:  
* Benign LinearSVC--validation set: 94.23%
* Malignant LinearSVC--validation set: 98.32%  
Furthermore, the prediction accuracy for the malignant class (attacks) obtained by the SVM model exceeds that for the benign class (normal traffic). Prioritizing the prediction accuracy of malignant traffic is advisable, as misclassifying benign instances as attacks is generally preferable to incorrectly classifying attacks as benign. These details are reflected in the confusion matrix presented in Table 3 and the classification report shown in Figure 4. Additionally, Figure 5 illustrates the AUC-ROC (Area Under the Curve-Receiver Operating Characteristic) curve for the validation set using the SVM model. The results obtained by the SVM model on the test set are as follows:  
* Benign class (LinearSVC)--test set accuracy: 94.23%
* Malignant class (LinearSVC)--test set accuracy: 98.35%The prediction percentage for the malignant class (attacks) obtained by the SVM model on the test set is higher than the prediction percentage for the benign class (normal traffic). This is because it is preferable to classify certain instances of benign traffic as attacks (malignant traffic) rather than the other way around. Furthermore, due to the imbalance between the two classes (malignant and benign) in the test set, a decrease in accuracy, precision, and F1-score can be observed for the malignant class (attacks), alongside a slight increase in these metrics for the benign class (normal traffic). These details are reflected in the confusion matrix presented in Table 4 and the classification report shown in Figure 6. Additionally, Figure 7 illustrates the AUC-ROC curve for the test set using the SVM model. We will continue with the RF results, starting with the validation set. The performance obtained by RF on this set is as follows:  
* Benign class (RF)--validation set accuracy: 99.99%
* Malignant class (RF)--validation set accuracy: 99.98%  
The prediction accuracy for the malignant class (attacks) achieved by the RF model on the validation set is nearly equal to that for the benign class (normal traffic). This contrasts with the SVM model, which exhibited a significantly higher prediction error compared to RF. These results can be observed in the confusion matrix presented in Table 5 and the classification report shown in Figure 8. Additionally, Figure 9 illustrates the AUC-ROC curve for the test set using the RF model. * Malignant class (RF)--test set accuracy: 99.98%  
Due to the class imbalance between malignant (attacks) and benign (normal traffic) instances in the test set, a slight decrease in precision and F1-score for the malignant class was observed, accompanied by a minor increase in metrics for the benign class. Nonetheless, the overall accuracy improved slightly from 0.999855 to 0.999894, as the algorithm had enough instances to identify attack patterns and enhance overall recall. These results can be observed in the confusion matrix presented in Table 6 and the classification report shown in Figure 10. Additionally, Figure 11 illustrates the AUC-ROC curve for the test set using the RF model. Figure 14: RF precision–recall curve for measuring the precision–recall. Figure 12: Comparing the accuracy of algorithms. Figure 13: SVM precision–recall curve for measuring the precision–recall. * Average precision--recall value for SVM--binary classification: 0.80
* Average precision--recall value for RF--binary classification: 1.00  
The results obtained by the SVM model on the test set, evaluated for all attacks collectively as well as for each individual attack, are summarized in Table 7.