# Unsupervised Machine Learning Techniques for Detecting PLC Process Control Anomalies
## 5 Results and Discussions
### Performance Evaluation
#### 5.2.1 Performance of OCSVM  
OCSVM's recall values on test sets 1, 2, 3, 4, and 5 are 90%, 87%, 86%, 81%, and 70%, respectively. Figure 6 illustrates the benefits of the proposed visualization as it reveals OCSVM's overall behavior on all test sets. A similar distribution of TP scores is  
Figure 6: OCSVM results of the normalized TP, TN, FP, and FN values on test sets 1–5.  
observed on all test sets, showing how OCSVM learned the TLIGHT system's normal behavior during training process. OCSVM found detecting normal traffic transitions involving vehicles' green light signals challenging, leading to high TP scores that lie along the decision boundaries in all test cases. Again, FN scores of test sets 1 and 3 involve the same data records being misclassified as anomalies. Moreover, FN scores of test sets 4 and 5 represent identical normal data records misclassified as anomalies. The aforementioned show the importance of the proposed visualization in this work as Figure 6 reveals OCSVM's true performance on each data record, which would not be possible with the traditional histogram approach [33].  
Although OCSVM's recall values decrease steadily from test sets 1 to 5, the normalized true positive scores are low in all test cases. In all test cases, OCSVM misclassifies over 30% anomalous instances as normal instances leading to high levels of FP in all test cases. In addition, Figure 6 shows that in all test cases in which OCSVM makes an accurate positive class (normal) prediction, the score (levels of confidence) of predictions are low and lie along the decision boundary. Although OCSVM detects several positive classes, it has a greater chance of misclassifying normal data points as anomalies because of the low confidence of the positive class prediction. Moreover, about 75% of correct positive class predictions occur close to the decisions boundary with low prediction confidence. The low prediction confidence makes OCSVM unstable for TLIGHT system anomaly detection. Figure 6 shows a histogram of normalized FP, FN, TP, and TN decision scores made by OCSVM on test sets 1-5.  
OCSVM learns the TLIGHT system's normal behavior during training with a 100% precision and 98% F1-score. The results substantiate the procedure this work adopts in conducting the experiment and recording data as it is similar to the training results in [13]. Overall, OCSVM performs best on test set 1. OCSVM performance on test sets 2 and 3 are similar with an F1-score of 84%. OCSVM has its least performance on test sets 4 and 5, with F1-scores of 71% and 68% respectively. OCSVM model is unable to detect over 20% of anomalies in test sets 4 and 5 because of the large proportion of anomalies consisting of timing bits anomalies. Therefore, OCSVM appears to be ineffective at detecting TLIGHT system errors consisting of system timing bits manipulation. Table 5 summarizes the performance of the OCSVM described here and the OCSVM in [13]. Table 5 shows that the OCSVM reported here and in [13] have similar training performance due to the datasets and underlying TLIGHT experiment being designed identically. Test sets 1-3 in this work are created to be the same size as the test sets in [13]; however, the exact nature and distributions of errors in [13] are unknown and may be the reason for the slight differences seen in the Table 5.