# Launching Adversarial Attacks against Network Intrusion Detection Systems for IoT
## 5 Evaluation and Discussion
### SVM Dataset Label Manipulation  
In Section 3.3, it was discussed that an increase in FNR would allow for a greater likelihood of the model not detecting malicious events. The FNR is inversely proportional to the recall score; albeit the FN count increases, the recall score will decrease. The recall metrics that are generated in Appendix F, Figure 11 have been extracted, and are seen in Figure 5. It is observed in Figure 11a that in targeted label flipping, the accuracy score falls at a quicker rate than the recall score in the case of targeted label flipping. This is because the number of FN is not increasing as quickly as the false positive rate. In the case of random label flipping, as observed in Figure 11b, the recall score is firmly related to the accuracy score. The strong correlation could mean that the accuracy is falling because the FNR increases more than the targeted experiment. Therefore, the outcome is that an adversary would have more success if they performed random label flipping. However, comparing these results to the works of [21, 37], it was found that one must flip labels with a large margin to distort the hyperplane significantly. The hypothesis suggested in Section 3.2 of flipping labels with a small margin to the hyperplane is false. To distort the hyperplane, labels' generation with large margins must be selected.  
Figure 4: (**a**) Confusion matrices of binary class perturbed with 1.0 epsilon. (**b**) Confusion matrices of multi-class perturbed with 1.0 epsilon.  
\begin{table}
\begin{tabular}{l c c c c c c c c c} \hline \hline
**Scoring** & \multicolumn{2}{c}{**Accuracy**} & \multicolumn{2}{c}{**Loss**} & \multicolumn{2}{c}{**Precision**} & \multicolumn{2}{c}{**Recall**} & \multicolumn{2}{c}{**F1**} \\ \hline \hline \multirow{2}{*}{Epsilon} & 0 & 1 & 0 & 1 & 0 & 1 & 0 & 1 & 0 & 1 \\ \hline \hline Binary Targeted & 0.996 & 0.927 & 0.016 & 0.151 & 0.996 & 0.895 & 0.996 & 0.563 & 0.996 & 0.690 \\ \hline \hline Binary Non-Targeted & 0.996 & 0.768 & 0.016 & 1.080 & 0.996 & 0.769 & 0.996 & 0.771 & 0.996 & 0.769 \\ \hline \hline Multi-Targeted & 0.956 & 0.421 & 0.045 & 1.764 & 0.952 & 0.312 & 0.957 & 0.493 & 0.955 & 0.382 \\ \hline \hline Multi-Not-Targeted & 0.956 & 0.141 & 0.045 & 2.403 & 0.952 & 0.153 & 0.957 & 0.249 & 0.955 & 0.189 \\ \hline \hline \end{tabular}
\end{table}
Table 5: Effect of Zero epsilon vs. 1 epsilon against the metrics using Fast Gradient Sign Method in targeted and non-targeted modes.  
Still, further concerns must be considered regarding the targeted label flipping attack against the SVM. The metrics show a decreasing trend as the percentage of label flips go up. However, it must be considered why the metric gradient is repetitively alternating from positive to negative. The complete effect of 0% and 50% label flip in Table 4 shows that the metrics all decrease, nevertheless in Appendix F, Figure A1 shows the alternating positive and negative metric gradients. This anomaly could be explained depending on how the hyperplane is adjusted after flipping \(X\%\) of labels. For example, labels with a margin of 1% over those flipped labels may be manipulating the result. Therefore, the alternating positive and negative metric gradients is an anomaly.  
Finally, the method of attack against the SVM must be considered. The threat model used against the SVM assumed that the adversary had access to the training dataset. The authors in [20] stated that if an adversary can manipulate the training dataset, the results should correspond with a strong negative impact on the metrics. Nevertheless, even though the accuracy and recall scores decreased, the metrics' amount could be considered insignificant. It took a large amount of data manipulation to substantially impact the metrics, using adversarial countermeasures such as statistical analysis, and this attack could be detected [21].