# LLM-Based Cyberattack Detection Using Network Flow Statistics
## 3 Experimental Results
While the latter two showed reasonable accuracy given their sample sizes, their performance was still impacted by their minority status compared to the overall dataset. This indicates that the primary challenge for these specific attack types lies in the severe data imbalance at the sub-category level. The limited availability of training data for some types of attacks hinders the capacity of LLMs to classify them accurately, leading to an increased number of false negatives or misclassifications, particularly for SQL Injection, Brute Force-Web and Brute Force-XSS. Appendix A contains and describes the confusion matrices for the CIC-IDS-2017 and BCCC-CIC-IDS-2017 datasets. \begin{table}
\begin{tabular}{c c c c c c c c c c c c c c c} \hline \hline
**Class** & **0** & **1** & **2** & **3** & **4** & **5** & **6** & **7** & **8** & **9** & **10** & **11** & **12** & **13** & **14** & **Misclassified** \\ \hline
0 & 532,635 & 19 & 0 & 0 & 14 & 1,420 & 5 & 0 & 0 & 0 & 3 & 0 & 0 & 0 & 0 & 0.27\% \\
1 & 4 & 115,231 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0\% \\
2 & 0 & 0 & 133,692 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0\% \\
3 & 0 & 0 & 0 & 86,975 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0\% \\
4 & 5 & 0 & 0 & 0 & 56,457 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0.01\% \\
5 & 72 & 0 & 0 & 0 & 0 & 32,049 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0.22\% \\
6 & 0 & 0 & 0 & 0 & 0 & 0 & 23,464 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0\% \\
7 & 1 & 0 & 0 & 0 & 0 & 0 & 0 & 8290 & 0 & 0 & 0 & 0 & 0 & 0 & 0.01\% \\
8 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 2057 & 0 & 0 & 0 & 0 & 0 & 0 & 0\% \\
9 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 346 & 0 & 0 & 0 & 0 & 0 & 0\% \\
10 & 7 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 106 & 7 & 2 & 0 & 0 & 13.11\% \\
11 & 3 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 42 & 1 & 0 & 0 & 8.7\% \\
12 & 5 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 1 & 7 & 4 & 0 & 0 & 76.47\% \\
13 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 3892 & 0 & 0\% \\
14 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 7869 & 0\% \\ \hline \hline \end{tabular}
\end{table} 
Table 8: Confusion matrix, with \(y\) as rows and \(\hat{y}\) as columns, for CSE-CIC-IDS2018. An initial analysis of the confusion matrix for the CSE-CIC-IDS2018 dataset was conducted, from which we extracted interesting information. The next step of our analysis was to study the effectiveness of our system using the aforementioned metrics, namely, accuracy, precision, recall, F-score, and some critical cybersecurity-specific metrics such as False-Positive Rate (FPR), False-Negative Rate (FNR), and Detection Latency. Without these metrics, we cannot verify whether our methodology is valid regardless of the dataset used. Table 9 displays all the results for all the selected metrics, using the previously studied CSE-CIC-IDS2018, as well as CIC-IDS-2017 and BCCC-CIC-IDS-2017. We obtained the best results when applying our methodology to the CIC-IDS-2017 dataset. We obtained 99.94% for all the metrics, which is 0.1% better than applying it to CSE-CIC-IDS2018. Regarding the augmented dataset, we obtained 99.9% for all the metrics applying our approach to BCCC-CIC-IDS-2017. Considering that this dataset contains a larger set of features, we obtained results that were 0.04% worse than those achieved with the original dataset, CIC-IDS-2017. Nonetheless, we can see that all the results are relatively consistent and present promising values according to the metrics selected for this study. On the other hand, the CIC-IDS-2017 also demonstrated an exceptionally low FPR of only 0.01%, indicating a minimal rate of false alarms, which is crucial for reducing alert fatigue in operational security centers. Furthermore, its FNR of 0.05% underscores the model's high effectiveness in detecting actual attacks, significantly reducing the risk of missed threats. For CSE-CIC-IDS2018, it recorded an FPR of 0.27% and an FNR of 0.02%. Although its FPR is higher than for CIC-IDS-2017, it remains within an acceptable range for many security operations.