# APT-Attack Detection Based on Multi-Stage Autoencoders
## 5 Evaluation
### Evaluation--Zero-Day-Attacks Test Dataset  
An important challenge for ML algorithms constitutes the evaluation of methods for scenarios with unknown attacks. These \"zero-day attacks\" do not correspond to trained application scenarios in boundary conditions. Analogous to the evaluations for the independent test dataset, Table 7 and 8 show numerical results analogous to the measures and metrics, as defined above. It should be noted that this dataset contains 1,593,831 entries, out of which 1,276,109 entries constitute benign data with 317,722 attack traces. In addition, the contamination rate equals 19.93% in this case. \begin{table}
\begin{tabular}{c c c c c c c} \hline \hline
**Method** & **Cont. Rate** & **TP** & **TN** & **FP** & **FN** & **TPR** & **TNR** \\ \hline AE & & 94,301 & 1,143,117 & 132,992 & 223,421 & 29.7\% & 89.6\% \\ AEC & & 81,252 & 1,236,751 & 39,358 & 236,470 & 25.6\% & 96.9\% \\ AE-CNN & & 80,846 & 1,221,599 & 54,510 & 236,876 & 25.4\% & 95.7\% \\ VAE & 19.93\% & 89,072 & 1,143,910 & 132,199 & 228,650 & 28.0\% & 89.6\% \\ VAE-Prob & & 133,394 & 1,049,943 & 226,166 & 184,328 & 42.0\% & 82.3\% \\ AE+VAE-Prob & & 133,182 & 1,137,382 & 138,727 & 184,540 & 41.9\% & 89.1\% \\ OCSVM & & 82,771 & 1,169,204 & 106,905 & 234,951 & 26.1\% & 91.6\% \\ \hline \hline \end{tabular}
\end{table} 
Table 7: Resulting confusion matrix for the detection of anomalies in the zero-day-attacks test dataset. Figure 6: Results for the probability-based variational autoencoder. In the case of the confusion matrix, it is noticeable that, with threshold values determined in the training, consistently similar TNRs are achieved as in the evaluation of the independent test data set. However, the TPRs are significantly lower, in some cases in the range of 20-30%. This can also be observed in the reference approach of classic ML. The same conclusion applies for the best possible autoencoder in unrealistic scenarios (AEC). Thus, it can be claimed that this dataset is characterized by significantly increased complexity and difficulty. The most remarkable result is the excellent performance of the probabilistic variation autoencoder (VAE-Prob) with a TPR of over 40%. The two-stage approach (AE+VAE) exhibits a comparable performance as well. Both observations can be regarded as remarkable results, especially when considering the significantly reduced runtimes (see final column in Table 8). Similar outcomes are also achieved for \(f_{1}\) values in the context of AE-VAE. In addition, the runtime of OCSVM is extremely high in this case. This means that it is analogous to the results for the independent test data sets. Another overview of the results provides the receiver operating curve (ROC) in Figure 7. This representation depicts the obtained threshold values for each algorithm. The red dots mark the optimal threshold values that are determined during the training. On the other hand, the black triangles indicate the position of optimal threshold values with regard to zero-day attacks. It should be noted that OCSVM does not achieve any meaningful results in this case. In addition, the results are even totally random with some threshold settings (brown curve). The autoencoder with custom loss AEC (orange), which cannot be used in practical operation despite its theoretical efficiency, also performs significantly worse than in the previous evaluation for the independent test data set. A qualitatively similar conclusion emerges for the other algorithms as well. Whereas AE-CNN performs in the weakest manner, AE and VAE achieve better results. However, it is noticeable that all optimised threshold values from training (red dots) can be found in a rather low TPR range. Here, too, corresponding adaptations and adjustments to threshold values can lead to significantly better detection rates in this scenario. This includes operators based on previous knowledge that are limited by the number of possible false detections--FPs.