# Analysis of Lightweight Feature Vectors for Attack Detection in Network Traffic
## 5 Results and Discussion
### Classification Performance  
Table 5 shows the classification results when using random forests, which are the supervised learners that obtained the best performances during experiments. We provide training and test cross-validates indices to show that the learner was robust enough and did not overfit, that is, it performed alike during training and test phases (for this reason, in order to make training and test processes comparable, test results are also shown in a cross-validated way.) Table 5 shows similar performances for the CAIA, Consensus, and UNSW vectors. The TA vector is considerably inferior and the AGM vector clearly outperforms its competitors. Among the CAIA, Consensus, and UNSW vectors, the UNSW vector performs a bit worse and the Consensus and CAIA vectors show equivalent performances. Results show that the approach suggested by the AGM vector might reveal relevant traits missed by analysis based on flow behaviors. At the expense of increasing computational costs, capturing source behaviors offers richer profiles, with new points of support for discriminating malicious activity from legitimate communications. Real applications might benefit from the combination of both analysis perspectives: Flow-profiling and source-profiling. Here, it should be noted that the analyzed dataset was generated by the IXIA PerfectStorm platform. With artificially generated data, it is possible that analysis methods are influenced by artifacts intrinsic to the data generation platform or its limitations. We consider this unlikely in general. Note that the UNSW dataset has been designed to be representative and comprehensive to test attack detection solutions; similarly, the IXIA platform has been specifically created for security  
\begin{table}
\begin{tabular}{c c c c c c c c} \hline
**Top** & **Feat. Vector** & **Train./Test** & **Accuracy** & **Precision** & **Recall** & **F1.score** & **Roc.auc** & **Param. *** \\ \hline
4 th & UNSW & training & 0.990 & 0.859 & 0.851 & 0.849 & 0.998 & _nsl = 4, mdp = 14_ \\  & & & \(\pm\)0.005 & \(\pm\)0.194 & \(\pm\)0.100 & \(\pm\)0.056 & \(\pm\)0.002 & \\  & & test & 0.989 & 0.849 & 0.851 & 0.849 & 0.998 & \\  & & & \(\pm\)0.006 & \(\pm\)0.200 & \(\pm\)0.098 & \(\pm\)0.059 & \(\pm\)0.002 & \\ \hline
2 nd/3 rd & CAIA & training & 0.990 & 0.861 & 0.860 & 0.854 & 0.998 & _msl = 4, mdp = 14_ \\  & & & \(\pm\)0.007 & \(\pm\)0.221 & \(\pm\)0.081 & \(\pm\)0.077 & \(\pm\)0.002 & \\  & & test & 0.989 & 0.851 & 0.843 & 0.839 & 0.998 & \\  & & & \(\pm\)0.007 & \(\pm\)0.229 & \(\pm\)0.097 & \(\pm\)0.075 & \(\pm\)0.003 & \\ \hline
2 nd/3 rd & Consensus & training & 0.990 & 0.863 & 0.856 & 0.853 & 0.998 & _msl = 4, mdp=_15 \\  & & & \(\pm\)0.007 & \(\pm\)0.216 & \(\pm\)0.078 & \(\pm\)0.074 & 0.002 & \\  & & test & 0.999 & 0.853 & 0.847 & 0.843 & 0.998 & \\  & & & \(\pm\)0.008 & \(\pm\)0.225 & \(\pm\)0.067 & \(\pm\)0.082 & \(\pm\)0.003 & \\ \hline
5 th & TA & training & 0.986 & 0.767 & 0.839 & 0.794 & 0.995 & _msl = 6, mdp = 13_ \\  & & & \(\pm\)0.010 & \(\pm\)0.236 & \(\pm\)0.064 & \(\pm\)0.106 & \(\pm\)0.004 & \\  & & test & 0.986 & 0.763 & 0.812 & 0.780 & 0.993 & \\  & & & \(\pm\)0.009 & \(\pm\)0.224 & \(\pm\)0.054 & \(\pm\)0.082 & \(\pm\)0.003 & \\ \hline
1 st & AGM & training & 0.995 & 1.000 & 0.940 & 0.968 & 0.990 & _msl = 1, mdp = 9_ \\  & & & \(\pm\)0.008 & \(\pm\)0.000 & \(\pm\)0.098 & \(\pm\)0.052 & \(\pm\)0.040 & \\  & & test & 0.987 & 0.946 & 0.876 & 0.908 & 0.984 & \\  & & & \(\pm\)0.008 & \(\pm\)0.130 & \(\pm\)0.057 & \(\pm\)0.046 & \(\pm\)0.015 and performance testing. However, dataset design conditions might be favoring AGM-based detection, since attacker hosts in the UNSW dataset are very active as a general rule. In any case, it would be very valuable to analyze additional modern, labeled datasets for network security in order to corroborate findings. Unfortunately, as discussed in Section 3, there is an alarming lack of validated labeled datasets that are representative and rigorous enough for research purposes.