# APT-Attack Detection Based on Multi-Stage Autoencoders
## 5 Evaluation  
### Evaluation Metrics  
In order to evaluate the applied ML methods in our experiments, we rely on existing performance metrics from this domain. Therefore, we rely on the ROC (receiver operating characteristic) curve for the subsequent analysis. This graphical plot depicts the performance of anomaly detection based on multiple threshold values. However, the respective overall performance measurement across all of ROC's threshold settings represents AUC-ROC (area under the ROC) curve. This metric is applied in ML for checking the performance of the classification model, as required in this paper. In general, the ROC curve depicts probabilities, whereas AUC measures how the ML model distinguishes between different classes. The higher the AUC, the better the model performs this task.  
In addition to ROC and AUC, the following metrics derived from the individual threshold values and based on the confusion matrix are commonly considered for evaluation: balanced accuracy, precision, recall and \(f_{1}\) score [43; 44].  
In practice, the basic evaluation measures (confusion matrix) represent the absolute number of correctly recognized anomalies (TP--true positives), the number of correctly recognized benign data (TN--true negatives), benign data wrongly recognized by the algorithm as anomalies (FP--false positives) and the wrongly overlooked by the algorithm anomalies (FN--false negatives). In order to compare data sets of different sizes, they are normalized according to the dataset size, i.e., the amount of benign data or anomalies (TP rate or TPR, TN rate or TNR). Subsequently, derived parameter values in the evaluation are accuracy (acc), balanced accuracy (balAcc), precision (prec or PPV), recall (rec, equal to TPR) and \(f_{1}\) score, as common in the literature.  
The \(f_{1}\) score, the most commonly used evaluation metric in similar applications, represents an evaluation measure between \"recall\" and \"precision\" in the form of a single number. The respective value can be further used depending on the area of application and desired statement. This equation is defined in the following manner:  
\[f_{1}=\frac{2\times\text{TP}}{2\times\text{TP}+\text{FP}+\text{FN}} \tag{3}\]  
Balanced accuracy (balAcc), in contrast to \"accuracy\", is particularly suitable for the evaluation of unbalanced datasets--datasets that exhibit large differences in the amount  
\begin{table}
\begin{tabular}{c c c c c c} \hline \hline
**Type/Value** & **AE** & **AE-CNN** & **VAE** & **VAE-Prob** & **AEC** \\ \hline Activation & selu & relu & relu & relu & relu \\ \hline Batch size & 64 & 128 & 128 & 524 & 64 \\ \hline Dropout rate & 0.3 & 0.3 & 0 & 0 & 0 \\ \hline Epochs & 100 & 100 & 100 & 100 & 100 \\ \hline Initializer & lecun\_n & lecun\_n & lecun\_n & lecun\_n & he\_n \\ \hline Regularizer & l1 & l2 & l1 & l1 & None \\ \hline CNN kernel size & - & 3 & - & - & - \\ \hline Hidden neurons & 10 & 14 & 14 & 16 & 10 \\ \hline Layer red. & True & True & False & True & True \\ \hline Learning rate & 0.01 & 0.001 & 0.01 & 0.01 & 0.001 \\ \hline Loss function & mse & mse & mse & mse & custom \\ \hline Conv. layers & - & 2 & - & - & - \\ \hline Dense layers & 5 & 2 & 4 & 3 & 4 \\ \hline Optimizer & Adam & Adamax & Adamax & Adadelta & Adamax \\ \hline \hline \end{tabular}
\end{table}
Table 4: Optimised parameters with hyperparameter optimisation.  
of positive and negative data, i.e., anomalies and benign data. It is defined as the average value of TPR and TNR. The equation for \"balanced accuracy\" is given as follows:  
\[bal\,Acc=\frac{\text{TPR}+\text{TNR}}{2} \tag{4}\]