# SCADA System Testbed for Cybersecurity Research Using Machine Learning Approach
## 6 Numerical Results  
In this section, we present the numerical results of the attacks described in Section 5.1. Figure 8 shows the results for the accuracy of the ML algorithms that were used.  
The accuracy represents the total number of correct predictions divided by the total number of samples (Equation (1)). As shown in Figure 8, considering the offline evaluations, Decision Tree and  
Figure 8: Accuracy results.  
Figure 7: Model evaluation.  
KNN have the best accuracy (100%) compared to other ML models. However, the difference in the accuracy is small among all trained models. In other words, all chosen ML algorithms performed well in terms of accuracy during the offline phase. During the online phase, Decision Tree, Random Forest, Naive Bayes and Logistic Regression show a small difference, hence, the performance of these algorithms in both phases (offline and online), are similar. The same does not apply to the KNN model. There was a significant difference between the online and offline phase which indicates that in practice, the KNN does not provide good accuracy.  
As shown in Table 5, our dataset is unbalanced. Therefore, accuracy is not the ideal measure to evaluate performance [33]. Other metrics are needed to compare the performance of the ML algorithms. Figure 9 shows the false alarm rate (FAR) results. The FAR metric is the percentage of the regular traffic which has been misclassified as anomalous by the model (Equation (2)).  
Regarding the offline and online evaluations, as shown in Figure 9, the Random Forest and Decision Tree models performed best, followed by the KNN model. These three models had the lowest false alarm percentages followed by the Logistical Regression and Naive Bayes. These lowest percentages mean that Random Forest, Decision Tree, and KNN perform better in detecting normal traffic. In our dataset, normal traffic is the dominant traffic; therefore, it is expected to have a low FAR value. This low FAR value could be due to the model's bias toward estimating the normal traffic perfectly, which is common in unbalanced datasets. Further, the clustering done in the Random Forest, Decision Tree and KNN models can be helpful, especially when dealing with two types of data having different network features.  
Figure 10 shows the results of the un-detection rate metric. The UND (Equation (3)) represents the percentage of the traffic which is an anomaly but is misclassified as normal (the opposite of the FAR). The traffic represented by this metric is more critical than the traffic represented by the FAR metric because, in this case, an attack can happen without being detected. Further, in our unbalanced dataset, the models are biased toward normal traffic and this metric would show how biased the models are.  
As shown in Figure 10, considering the offline performance results, the percentage of the UND is small for the Naive Bayes, Logistic Regression, and KNN models, and zero for the Decision Tree and Random Forest models. That is, all algorithm shows excellent performance on this critical metric. However, considering the online performances, KNN model had the worst performance, which was very different to the offline evaluation. The same did not happen to the other models, and their online performances are very close to their offline performance. This excellent performance shows that the features selected in this work are also very good as they were able to detect attacks even in an unbalanced dataset.  
Figure 9: False alarm rate results.