# Securing Network Traffic Classification Models against Adversarial Examples Using Derived Variables
## 6 Results and Analysis
### Comparative Analysis  
The experimental results are presented below to provide a concise summary of the findings. The performance of various ML models in the absence of AEs is summarized in Table 4. The evaluation metrics of accuracy (Acc), precision (Prec), recall (Rec), and F1-score (F1) were used to assess model performance. AdverNet, which is a feed-forward neural network specifically employed to craft AEs, exhibited accuracy of 98%, while the DT, RF, and KNN models showed exceptional performance, achieving accuracy scores of 100% and consistent metrics at 99%. Additionally, the training curves of AdverNet on the baseline dataset are depicted in Figure 5.  
Furthermore, as detailed in Table 5, we present a comparative analysis of baseline performance using the CSE-CIC-IDS2018 dataset. Notably, our approach consistently achieved exceptional levels of accuracy and other performance metrics, surpassing established baseline models. This table underscores the pronounced superiority of DVars on clean examples, thus reaffirming their substantial influence in the field of network traffic classification research. Additionally, a comparative study examining both the attack success rate and defense effectiveness on the leading models considered within the same dataset is presented in Tables 6 and 7, respectively. The proposed defense outperformed the \begin{table}
\begin{tabular}{c c c c c} \hline  & **Acc (\%)** & **Prec (\%)** & **Rec (\%)** & **F1 (\%)** \\ \hline AdverNet & 0.98 & 0.98 & 0.98 & 0.98 \\ DT & 1.00 & 1.00 & 1.00 & 1.00 \\ RF & 0.99 & 0.99 & 0.99 & 0.99 \\ KNN & 0.99 & 0.99 & 0.99 & 0.99 \\ \hline \end{tabular}
\end{table} 
Table 4: Baseline results of the target models in accuracy, precision, recall, or F1-score. \begin{table}
\begin{tabular}{c c} \hline
**Model** & **Parameters** \\ \hline AdvNet & Layer 1 = 128, Layer 2 = 64, Layer 3 = 3, \\  & activation = Relu, optimizer = adam, \\  & output\_layer\_activation = softmax, epochs = 50 \\ DT & Criterion = gini, max\_depth = 12 \\ RF & Number of estimators = 170, random\_state = 4, \\  & min\_sample\_split = 5 \\ KNN & Number of neighbors = 10, \\  & distance metric = Euclidean \\ JSMA & \(\theta=1.0\), \(\gamma=0.1\), clip min = 0.0, clip max = 1.0 \\ \hline \end{tabular}
\end{table} 
Table 3: Hyper-parameters of the studied models and attack method. referenced studies, except for the removal of altered features (RAF), which was examined with a self-validated attack. The results in Table 8 underscore the effectiveness of our defense strategy against adversarial attacks with different perturbation intensity. In the absence of perturbations, all models (DT, RF, KNN) performed exceptionally well, reflecting their robustness. As the perturbation intensity increased, the model performance gradually diminished. Despite this, KNN consistently demonstrated a higher degree of defense effectiveness, outperforming DT and RF, particularly at low and medium perturbation levels. This highlights the effectiveness of our defense mechanism in enhancing model resilience to adversarial attacks, with KNN emerging as the most reliable choice for such scenarios. The results obtained when no AEs were present in the dataset are shown in (Figure 6). The figure shows Acc, Prec, Rec, and F1 with respect to each class in the dataset for different classification models, including DT, RF, KNN, and AdverNet. As can be seen in this figure, all the models performed well in classifying the network flows, achieving the best accuracy scores across all classes present in the dataset. \begin{table}
\begin{tabular}{c c c c c} \hline \hline
**Work** & **Attack** & **Acc (\%)** & **Rec (\%)** & **F1 (\%)** \\ \hline [PERSON] et al. [6] & Self & - & 0.48 & - \\ [PERSON] et al. [58] & JSMA & 0.84 & 0.57 & 0.59 \\ [PERSON] et al. [59] & C\&W & 0.81 & 0.81 & 0.83 \\ [PERSON] et al. [60] & JSMA & 0.94 & - & - \\ Ours & JSMA & 0.45 & 0.48 & 0.44 \\ \hline \hline \end{tabular}
\end{table} 
Table 6: Comparison of attack success rate of the leading models using the CSE-CIC-IDS2018 dataset.