# Dual Auto-Encoder GAN-Based Anomaly Detection for Industrial Control System
## 5 Experimental Evaluation
### Sensitivity Analysis of the Hyper-Parameter \(\gamma\)  
The first experimental objective is to verify the sensitivity of the hyper-parameter \(\gamma\) and find a stable region for different ICS datasets. The hyper-parameter \(\gamma\) is defined to measure the ratio of the marginal distribution score (Formula (19)) to the overall anomaly score (Formula (20)) for a test sample. The value range of hyper-parameter \(\gamma\) is [0; 1]. The larger the value, the larger the proportion of the marginal distribution score in the overall anomaly score. This further means that a test sample with high marginal distribution score is more likely to be judged as an outlier. To get the stable region of the hyper-parameter \(\gamma\), we performed two experiments based on four evaluation metrics of ACC, Rec, FPR, and F1. (1) The first experiment uses DS2 OS as the experimental data and uses the training set and test set given in Section 5.1 to train and optimize the DAGAN model. Because the DS2 OS dataset contains 7 kinds of attacks, 7 test sets consisting of 30 abnormal samples and 50 normal samples are established, each test set contains only one attack and is different from other test sets. By gradually increasing \(\gamma\) from 0 to 1, we try to find the stable region of parameter \(\gamma\) by observing the change of the average value of 7 test sets on 4 evaluation metrics, as shown in Figure 6a. (2) Similar to the first experiment, the second  
\begin{table}
\begin{tabular}{c c c} \hline
**Algorithm** & **Full Name** & **Implement** \\ \hline SVM [15] & Anomaly-based Intrusion Detection in Industrial Data with SVM & Python \\  & and Random Forests. & Python \\ GMM [11] & Unusual Activity and Anomaly Detection in Surveillance Using & Python \\  & GMM-KNN Model. & \\ AE [27] & Anomaly Detection for Industrial Control System based on & \\  & Auto-encoder Neural Network. & \\  & FenceGAN [4] & Fence GAN: Towards Better Anomaly Detection. & Python \\ GANomaly [5] & GANomaly: Semi-Supervised Anomaly Detection via Adversarial & \\  & Training. & \\ DAGAN & Dual Auto-Encoder GAN-Based anomaly detection for industrial & \\  & control system. & \\ \hline \end{tabular}
\end{table}
Table 1: Comparison algorithms.  
experiment uses SWaT dataset to train and optimize the DAGAN model. Additionally, 4 test sets consisting of 20 abnormal samples and 40 normal samples are established. The experimental results are shown in Figure 5(b).  
Figure 6 plots the ACC, Rec, FPR and F1 trend lines of the DAGAN model on the DS2 OS and SWaT datasets under different hyper-parameters \(\gamma\). From the figure, we can find the following observations. (1) As the hyper-parameter \(\gamma\) increases from 0 to 0.3, the anomaly detection performance (ACC, Rec, FPR and F1) of our DAGAN model improves rapidly. For example, for the SWaT dataset, the FPR value is only 0.09 at \(\gamma\) = 0.1, but the FPR value is 0.04 at \(\gamma\) = 0.3. (2) As hyper-parameter \(\gamma\) increases from 0.5 to 1.0 (or from 0.6 to 1.0), the ACC, Rec, FPR and F1 performance of our DAGAN model degrades rapidly on the DS2 OS dataset (or on the SWaT dataset). Such as, for DS2 OS dataset, the F1 value is 0.87 at \(\gamma\) = 0.6, but the F1 value is only 0.8 at \(\gamma\) = 0.8. (3) As hyper-parameter \(\gamma\) increases from 0.3 to 0.5 (or from 0.3 to 0.6), the performance of our model varies slightly and is relatively stable on the DS2 OS dataset (or on the SWaT dataset). For example, the Rec value on the SWaT dataset is almost the same at \(\gamma\) = 0.3 and \(\gamma\) = 0.4, both around 0.9. Additionally, the ACC value on the DS2 OS dataset is almost the same at \(\gamma\) = 0.4 and \(\gamma\) = 0.5, both around 0.92.  
In summary, the [0.3-0.6] is a stable region of hyper-parameters \(\gamma\) for the different ICS datasets. In later experiments, the hyper-parameter \(\gamma\) was set to 0.4.