# Dual Auto-Encoder GAN-Based Anomaly Detection for Industrial Control System
## 5 Experimental Evaluation
### Performance Analysis of Anomaly Detection  
The third experimental objective is to verify the anomaly detection performance of our DAGAN algorithm, and to further compare the GANomaly and DAGAN models. In this experiment, SVM, GMM, AE, FenceGAN, GANomaly are used as competitors, the ACC, Rec, FPR and F1 are used as evaluation metrics, and DS2 OS and SWaT are chosen as the experimental datasets. For the DS2 OS dataset, 7 test sets are established. Each test set consists of 30 abnormal samples and 50 normal samples, and each test set contains only one attack that is different from other test sets. For SWaT dataset, 4 test sets are established, and each test set consists of 20 abnormal samples and 40 normal samples. Figure 9 plots the anomaly detection accuracy of 6 models on the 7 test sets of the DS2 OS, where Figure 9a plots the ACC results, Figure 9b plots the Rec results, Figure 9c plots the FPR results, and Figure 9d plots the F1 results. In the figure, the green triangle indicates that the detection accuracy on this verification set is high, indicating that the attack is easy to detect, abbreviated. The green triangle indicates that the detection accuracy in the test set is high, which implies that the attack in this test set is easy to be detected, simply called easy-to-identify attack. On the contrary, the red triangle indicates that the detection accuracy in the test set is low, which implies that the attack in this test set is difficult to be detected, simply called hard-to-identify attack. Form the figure, some observations are obtained. (1) For four metrics (ACC, Rec, FPR and F1), the 6 models have different performances, and the trend lines of these model have surged drastically. Comparing the 6 models, DAGAN, GANomaly, FenceGAN, and AE have the best anomaly detection  
results, SVM and GMM are the worst. Moreover, the stability of our DAGAN model is better than other models. (2) For easy-to-identify attacks with green triangles (DOS and DP), the detection accuracy of the DAGAN, GANomaly, FenceGAN and AE models is very high (The average value of ACC, Rec, or F1 is close to 0.95), and the performance of the four models is very close. (3) For hard-to-identify attacks with red triangles (MC, MO and WS), there are differences between the detection accuracy of the DAGAN, GANomaly, FenceGAN and AE models, and our model has a slight advantage. (4) Focusing only on the GANomaly and DAGAN models, the performance of the DAGAN model is better than that of the GANomaly model, especially for hard-to-detect attacks. Figure 10 further presents the anomaly detection performance of 6 models on 4 test sets of the SWaT from ACC, Rec, FPR, and F1 metrics. In the figure, the 6 models have achieved good results on different test sets, and the average values of the ACC, Rec and F1 are greater than 0.82. However, the performance of these 6 models varies greatly, with better performance on some test sets and poor performance on other test sets. More specifically, we can get the following observations. (1) For the ACC, the anomaly detection accuracy of the 6 models on the 4 test sets fluctuate sharply, and the stability is very poor. For example, the AE model performs poorly on SSMP and MSMP, but performs well on MSSP. Comparing these 6 models, DAGAN, GANomaly, FenceGAN and AE models have the best performance and stability. The SVM and GMM models have the worst performance and poor stability. (2) For the Rec, the curves of these 6 models are very uneven. Among them, DAGAN and GANomaly are the best, FenceGAN and AE are second, SVM and GMM are the worst. (3) For the FPR, the DAGAN and FenceGAN have better stability than other models. Moreover, for the 6 models, the advantage of our DAGAN model on hard-to-identify attack (red triangle) is stronger than on easy-to-identify attack (green triangle). (4) For the F1, the 6 models are more stable than the Rec and FPR metrics. Comparing the 6 models, DAGAN, GANomaly and FenceGAN models have better performance than the AE, GMM and SVM models.