# Adapative Perturbation Patterns: Realistic Adversarial Learning for Robust Intrusion Detection
## 4 Experimental Evaluation
### IoT Scenario Results  
In the IoT network scenario, the adversarial cyber-attack examples were generated using the original flows of the IoT-23 dataset. The analysis performed for the previous scenario was replicated to provide similar assessments, including the potential alternatives of the current literature: JSMA and OnePixel. Figure 6: Untargeted attack accuracy of Enterprise network scenario. Figure 7: Untargeted attack F1-Score of Enterprise network scenario. The randomly selected flow for the assessment of example realism had the 'DDoS' class label, which corresponds to a distributed denial-of-service attack performed by the malwares recorded in the IoT-23 dataset. A2 PM replaced the encoded categorical features of the connection state and history with another valid combination, already used by other original flows of the 'DDoS' class. Instead of an incomplete connection (OTH) with a bad packet checksum (BC), it became a connection attempt (S0) with a Synchronization flag (SYN). Hence, the generated network flow example remained valid and compatible with its intended malicious purpose, achieving realism. As in the previous scenario, both JSMA and OnePixel generated unrealistic examples. Besides the original OTH, both methods also increased the value of the feature representing an established connection with a termination attempt (S3). Since a flow with simultaneous OTH and S3 states is neither valid nor coherent with the cyber-attack's purpose, the methods remain inadequate alternatives to A2 PM for tabular data. In addition to the states, JSMA also assigned a single flow to two distinct communication protocols, transmission control protocol (TCP) and Internet control message protocol (ICMP), which further evidenced the inconsistency of the created data perturbations. Table 6 provides an overview of the modified features, with '-' indicating an unperturbed value. Regarding the targeted attacks, A2 PM caused much slower declines than in the previous scenario. The accuracy of the regularly trained MLP only started being lower than 50% at iteration 43, and RF stabilized with approximately 86%. These scores evidenced the decreased susceptibility of both classifiers, especially RF, to adversarial examples targeting the 'Benign' class. Furthermore, with adversarial training, the models were able to preserve even higher rates during an attack. Even though many examples still evaded MLP detection, the number of malicious flows predicted to be benign by RF was significantly lowered, which enabled it to keep its accuracy above 99%. Hence, the latter successful detected most cyber-attack variations (Figure 8). \begin{table}
\begin{tabular}{c c c c c} \hline \hline
**Feature** & **Original Value** & **A2 PM Value** & **JSMA Value** & **OnePixel Value** \\ \hline Connection state & ‘OTH’ & ‘S0’ & ‘OTH’ + ‘S3’ & ‘OTH’ + ‘S3’ \\ Connection history & ‘BC’ & ‘SYN’ & – & – \\ Communication protocol & ‘TCP’ & – & ‘TCP’ + ‘ICMP’ & – \\ \hline \hline \end{tabular}
\end{table} 
Table 6: Modified features of an adversarial ‘DDoS’ example. Figure 8: Targeted attack accuracy of IoT network scenario. The untargeted attacks iteratively caused small decreases of both metrics. Despite RF starting to stabilize from the fifth iteration forward, MLP continued its decline for an additional 48% of accuracy and 17% of macro-averaged F1-Score. This difference in both targeted and untargeted attacks suggests that RF, and possibly tree-based algorithms in general, have a better inherent robustness to adversarial examples of IoT network traffic. Unlike in the previous scenario, adversarial training did not provide considerable improvements. Nonetheless, the augmented training data still contributed to the creation of more adversarially robust models because they exhibited fewer incorrect class predictions throughout the attack (Figures 9 and 10). A time consumption analysis was also performed, to further analyze the scalability of A2 PM on relatively common hardware.