# Characterizing the Impact of Data-Damaged Models on Generalization Strength in Intrusion Detection
## 6 Discussion  
The scope of this investigation has led to a substantial set of results. In the results (Sections 4 and 5), the classification results for all attack classes as well as the trends observed from the visualizations are described in detail. Even though both sections end with intermediate conclusions, they remain dense. This section was included to give a straightforward view of the results and their implications. Table 3 shows the best three models per attack class from both a baseline perspective (B rows) and a generalized performance perspective (G rows). What is best is determined by the ranking of the models based on an equally weighted combination of balanced accuracy, F1-score and use of training data (less is better). The numeric columns (except for reduction) are percentages with a maximum of 100.  
Several points stand out in this table. First the baseline scores for all classes except infiltration (due to poor representation in CIC-IDS2017) are extremely high. Second and most importantly, pretrained models generalized well to the classes with clear network footprints such as bruteforce, L7-DoS, DDoS and botnet to some extent. These results are without any additional training and most often achieved by models that had little access to training data (\(\%\) training at or below \(1\%\), further broken down into one third training-two thirds validation). Third, although tree-based models typically have the highest baseline scores, the best generalizing models are not always tree-based. There are, however, more than enough tree-based models that do have great general performance (hence, they did not overfit) so it is possible. This is mainly interesting from a model interpretability perspective. Fourth and finally, the discrepancy between the baseline performance and the general performance for the web attacks would go unnoticed in most analyses, erroneously concluding that the models perform well on the class. This conclusion also applies to the global 2-class models.  
One crucial remark about Table 3 is that it obfuscates whether the models had a stable, generalized performance. This is most often not the case (as shown in the detailed results in Section 5). Improvements that guarantee stable, general class-level models after training should be sought after. These improvements could come from changes in data preparation and model selection based on generalization potential (either during or after training) or algorithmic modifications that improve robustness.  
\begin{table}
\begin{tabular}{c c c c c c c c c c} \hline \hline \multirow{2}{*}{**B/G**} & \multirow{2}{*}{**Class**} & \multirow{2}{*}{**Algorithm**} & \multicolumn{2}{c}{**Balanced Acc.**} & \multirow{2}{*}{**F1**} & \multirow{2}{*}{**Precision**} & \multirow{2}{*}{**Recall**} & \multirow{2}{*}{**Scaling**} & \multirow{2}{*}{**Reduction**} & \multirow{2}{*}{\(\%\)**Train**} \\ \hline \multirow{3}{*}{B} & \multirow{3}{*}{0.Bruteforce} & gradboost & 99.84 & 99.06 & 98.40 & 99.73 & No & 0 & 0.5 \\  & & extratree & 99.63 & 99.24 & 99.21 & 99.28 & Z & 0 & 0.5 \\  & & extratree & 99.64 & 99.63 & 99.97 & 99.28 & Z & 0 & 1.0 \\ \hline \multirow{3}{*}{G} & \multirow{3}{*}{0.Bruteforce} & xgboost & 100 & 100 & 100 & 100 & No & 0 & 0.5 \\  & & xgboost & 99.97 & 99.95 & 99.90 & 100 & No & 0 & 1.0 \\ \cline{1-1}  & & gradboost & 99.06 & 98.38 & 96.81 & 100 & MinMax & 0 & 0.1 \\ \hline \hline \end{tabular}
\end{table}
Table 3: Classification metrics for the best 3 models per attack class, both for baseline (B) and generalized (G) classification, with the mention of the preprocessing parameters.