## 5 Experimentation: UGR'16 as a Case Study
### Results and Discussion  
In this subsection, the obtained results are introduced both numerically, in Table 5, and graphically, in Figure 3. To get statistical confidence in our conclusions, tests were performed to assess statistical significance on the results achieved by the ML models tested. After analyzing the recommendations given in [44], the authors decided to apply pairwise Wilcoxon signed-rank test comparisons among all algorithms for all attack classes and accuracy metrics. The obtained \(p\)-values were corrected with Holm post-hoc method for multi-comparisons, and a 95% confidence level was considered. The differences in the presented accuracy of the models are statistically significant in all cases, except for:  
* DoS: Precision (SVC-RBF vs SVC-L), Recall (RF vs SVC-L), F1 (LR vs SVC-L, RF vs SVC-L, and LR vs RF), and AUC (LR vs SVC-L;
* Botnet: Precision (LR vs SVC-L), Recall (LR vs SVC-L and LR vs SVC-RBF), F1 (LR vs SVC-L and LR vs SVC-RBF), AUC (LR vs SVC-L and LR vs SVC-RBF);
* Scan: Recall (LR vs SVC-L, RF vs SVC-L and LR vs RF), F1 (RF vs SVC-L), AUC (LR vs SVC-L and RF vs SVC-L);
* Spam: Recall (LR vs SVC-L). We also applied the Friedman test to corroborate with 95% statistical confidence that there are significant differences in the accuracy of the models for all considered attacks and metrics. In general terms, all models offer good classification accuracy for those attacks synthetically generated (_DoS_, _Botnet_, _Scan_). However, a lower performance rate is observed when classifying real attacks (_Spam_) and real network traffic (_Background_). This behaviour was expected since real-network and real-attack traffic patterns are hard to model. Thus, existing or intentionally developed tools are not able to fully mimic such kind of real behaviours. \begin{table}
\begin{tabular}{c c c c c c} \hline \hline \multirow{2}{*}{**Model**} & \multirow{2}{*}{**Class**} & \multicolumn{4}{c}{**PM**} \\  & & **P** & **R** & **F1** & **AUC** \\ \hline \multirow{8}{*}{LR} & Background & 0.814 & 0.919 & 0.863 & 0.775 \\  & Dos & 0.933 & 0.915 & 0.923 & 0.957 \\  & Botnet & 0.965 & 0.891 & **0.926** & **0.945** \\  & Scan & 0.801 & 0.916 & 0.852 & 0.957 \\  & Spam & 0.797 & 0.606 & 0.688 & 0.764 \\ \cline{2-6}  & _Weighted avg._ & _0.810_ & _0.812_ & _0.805_ & _0.776_ \\ \hline \multirow{8}{*}{RF} & Background & 0.885 & 0.906 & 0.921 & 0.871 \\  & Dos & 0.973 & 0.884 & 0.925 & 0.942 \\  & Botnet & 0.977 & 0.922 & **0.948** & **0.961** \\  & Scan & 0.932 & 0.925 & 0.928 & 0.962 \\  & Spam & 0.917 & 0.749 & 0.824 & 0.857 \\ \cline{2-6}  & _Weighted avg._ & _0.897_ & _0.887_ & _0.888_ & _0.868_ \\ \hline \multirow{8}{*}{SVC-RBF} & Background & 0.839 & 0.937 & 0.885 & 0.810 \\  & Dos & 0.960 & 0.831 & 0.889 & 0.915 \\ \cline{1-1}  & Botnet & 0.972 & 0.886 & **0.927** & **0.943** \\ \cline{1-1}  & Scan & 0.941 & 0.536 & 0.652 & 0.768 \\ \cline{1-1}  & Spam & 0.824 & 0.638 & 0.717 & 0.790 \\ \cline{1-1} \cline{2-6}  & _Weighted avg._ & _0.837_ & _0.832_ & _0.827_ & _0.806_ \\ \hline \multirow{8}{*}{SVC-L} & Background & 0.819 & 0.93 & 0.871 & 0.785 \\  & Dos & 0.957 & 0.898 & 0.926 & 0.948 \\ \cline{1-1}  & Botnet & 0.968 & 0.899 & **0.932** & **0.949** \\ \cline{1-1}  & Scan & 0.944 & 0.91 & 0.926 & 0.955 \\ \cline{1-1}  & Spam & 0.829 & 0.61 & 0.703 & 0.773 \\ \cline{1-1} \cline{2-6}  & _Weighted avg._ & _0.826_ & _0.821_ & _0.815_ & _0.785_ \\ \hline \hline \end{tabular}
\end{table} 
Table 5: Average test performance results obtained by the different ML models considered. Specifically, RF provides the best results for almost all classes, followed by SVC-RBF. Note that RF is also overcoming the rest of the ML models tested when considering the weighted average of each performance metric. In this sense, RF is also a robust model for NIDSs when dealing with unbalanced classification where attack samples are considerably less than those found in the majority class, which might be a consequence of its internal ensemble classification procedure.