# Adapative Perturbation Patterns: Realistic Adversarial Learning for Robust Intrusion Detection
## 4 Experimental Evaluation
### Attacks and Evaluation Metrics  
A2 PM was applied to perform adversarial attacks against the fine-tuned models for a maximum of 50 iterations, by adapting to the data of the holdout evaluation sets. The attacks were untargeted, causing any misclassification of malicious flows to different classes, as well as targeted, seeking to misclassify malicious flows as the 'Benign' class. To perform a trustworthy evaluation of the impact of the generated examples on a model's performance, it was essential to select appropriate metrics. The considered metrics and their interpretation are briefly described below [42; 43].  
Accuracy measures the proportion of correctly classified samples. Even though it is the standard metric for classification tasks, its bias towards the majority classes must not be disregarded when the minority classes are particularly relevant to a classification task, which is the case of network-based intrusion detection [44]. For instance, in the Enterprise scenario, 77% of the samples have the 'Benign' class label. Since A2 PM was configured to not generate examples for that class, even if an adversarial attack was successful and all generated flows evaded detection, an accuracy score as high as 77% could still be achieved. Therefore, to correctly exhibit the misclassifications caused by the performed attacks, the  
\begin{table}
\begin{tabular}{c c} \hline \hline
**Parameter** & **Value** \\ \hline Splitting Criteria & Gini Impurity \\ Number of Trees & 100 \\ Maximum Depth of a Tree & 32 \\ Minimum Samples in a Leaf & 2 \\ Maximum Features & \(\sqrt{\text{Number of Features}}\) \\ Class Weights & Balanced \\ \hline \hline \end{tabular}
\end{table}
Table 4: Summary of random forest configuration.  
\begin{table}
\begin{tabular}{c c} \hline \hline
**Parameter** & **Value** \\ \hline Objective Loss & Categorical Cross-Entropy \\ Optimizer & Adam Algorithm \\ Learning Rate & 0.001 \\ Maximum Epochs & 50 \\ Class Weights & Balanced \\ \hline \hline \end{tabular}
\end{table}
Table 3: Summary of multilayer perceptron configuration.  
accuracy of a model was calculated using the network flows of all classes except 'Benign'. This metric can be expressed as:  
\[Accuracy=\frac{TP+TN}{TP+TN+FP+FN} \tag{5}\]  
where \(TP\) and \(TN\) are the number of true positives and negatives, correct classifications, and \(FP\) and \(FN\) are the number of false positives and negatives, misclassifications.  
Despite the reliability of accuracy for targeted attacks, it does not entirely reflect the impact of the performed untargeted attacks. Due to their attempt to cause any misclassification, their impact across all the different classes must also be measured. The F1-Score calculates the harmonic mean of precision and recall, considering both false positives and false negatives. To account for class imbalance, it can be macro-averaged, which gives all classes the same relevance. This is a reliable evaluation metric because a score of 100% indicates that all cyber-attacks are being correctly detected and there are no false alarms. Additionally, due to the multiple imbalanced classes present in both datasets, it is also the most suitable validation metric for the employed fine-tuning approach. The macro-averaged F1-Score is mathematically defined as:  
\[\text{Macro-averaged F1}-\text{Score}=\frac{1}{C}*\sum_{i=1}^{C}\frac{2*P_{i}*R _{i}}{P_{i}+R_{i}} \tag{6}\]  
where \(P_{i}\) and \(R_{i}\) are the precision and recall of class \(i\), and \(C\) is the number of classes.