# APT-Attack Detection Based on Multi-Stage Autoencoders
## 5 Evaluation
### Evaluation-Independent Test Dataset  
This section discusses the obtained evaluation results for the detection of anomalies in the independent test dataset, as shown in Table 5 and 6. In total, this dataset contains 1,637,230 entries, out of which 1,334,003 entries represent benign data and 303,227 are attack traces. The corresponding contamination rate equals 18.52%. With respect to the TP and TN rates, all algorithms exhibit good and practically useful results. However, AEC stands out as the theoretically best possible result. Unfortunately, this is only the case for unrealistic and practically irrelevant scenarios. In addition, AECNN and VAE achieve a slight improvement compared to the implemented standard ML method (OCSVM). As expected, the results of the purely autoencoder-based methods are below those of the variation autoencoder in terms of runtime. In any case, the OCSVM can be attested to have a \"negative\" runtime performance. It is important to mention that all values relate to a very special, selected working point during the process. All such points were determined during training in the course of the validation. A better assessment of the performance behaviour can be obtained from ROC curves, which are shown in Figure 4. All relevant TN and TP rates are plotted on the x and y axis, respectively. They include associated rates for different algorithms, shown in color. The red dots mark the optimal threshold values determined in the course of the training, whereas the black triangles show the position of the optimal threshold values with regard to the independent test data. The dotted line represents the performance of a \"random\" anomaly detector. The depiction demonstrates the superb efficiency of the normal autoencoder AE (blue), sampling-based autoencoder--VAE (red)--and the probabilistic autoencoder variant VAE-Prob (violet), compared to the standard OCSVM (brown). The same behaviour, however, is not pronounced for the convolution-based autoencoder AE-CNN (green) and the clearly superior, practically unusable--but theoretically best possible--autoencoder with custom loss-AEC (orange). \begin{table}
\begin{tabular}{c c c c c c c} \hline \hline
**Method** & **Balanced Accuracy** & **Accuracy** & **Precision** & **Recall** & \(f_{1}\)**Value** & **Running Time (s)** \\ \hline AE & 0.870 & 0.872 & 0.609 & 0.867 & 0.716 & 9.56 \\ AEC & 0.949 & 0.968 & 0.908 & 0.919 & 0.913 & 4.56 \\ AE-CNN & 0.797 & 0.905 & 0.817 & 0.625 & 0.708 & 14.17 \\ VAE & 0.808 & 0.871 & 0.635 & 0.709 & 0.670 & 243.04 \\ VAE-Prob & 0.847 & 0.824 & 0.515 & 0.884 & 0.650 & 286.31 \\ AE+VAE-Prob & 0.873 & 0.869 & 0.600 & 0.880 & 0.713 & 93.25 \\ OCSVM & 0.801 & 0.882 & 0.685 & 0.672 & 0.679 & 44,305.81 \\ \hline \hline \end{tabular}
\end{table} 
Table 6: Results for the detection of anomalies in the independent test dataset. \begin{table}
\begin{tabular}{c c c c c c c c} \hline \hline
**Method** & **Cont. Rate** & **TP** & **TN** & **FP** & **FN** & **TPR** & **TNR** \\ \hline AE & & 262,865 & 1,165,452 & 168,551 & 40,362 & 86.7\% & 87.4\% \\ AEC & & 278,736 & 1,305,643 & 28,360 & 24,491 & 91.9\% & 97.9\% \\ AE-CNN & & 189,636 & 1,291,406 & 42,597 & 113,591 & 62.5\% & 96.8\% \\ VAE & 18.52\% & 215,020 & 1,210,259 & 123,744 & 88,207 & 70.9\% & 90.7\% \\ VAE-Prob & & 268,052 & 1,081,114 & 252,889 & 35,175 & 88.4\% & 81.0\% \\ AE+VAE-Prob & & 266,852 & 1,155,967 & 178,036 & 36,375 & 88.0\% & 86.7\% \\ OCSVM & & 203,878 & 1,240,254 & 93,749 & 99,349 & 67.2\% & 93.0\% \\ \hline \hline \end{tabular}
\end{table} 
Table 5: Resulting confusion matrix for the detection of anomalies in the independent test dataset. Curiously, another interesting assumption can also be derived from the representation. The combined, two-stage approach--although very attractive in terms of the runtime performance (see Table 6)--cannot fully exploit the qualitative performance advantages in comparison to the normal autoencoder in the scenario examined.