# LLM-Based Cyberattack Detection Using Network Flow Statistics
## 3 Experimental Results  
The experiments were performed in an environment with software that is commonly used for data science. We selected Python 3.8.17 to develop the experiments. All experiments were performed on the supercomputer of the University of Cadiz. It contains several NVIDIA A100 GPUs with 40 GB of GPU memory and 256 GB of RAM, each node with 2 CPUs and each with 64 cores. Moreover, it contains a number of nodes with more memory, more concretely with 1 TB each node. The hyperparameter optimization process led us to select the best value for the hyperparameter of the maximum number of input tokens. Using the best configuration for the training step, we performed a fine-tuning task using the stratified fivefold CV approach, which executes five different fine-tuning processes, each with its related pair of training and validation subsets. Each execution generates a snapshot that performs the classification between benign and malignant data and, in the second case, directly obtains the type of attack between the 14 types contained in the CSE-CIC-IDS2018 dataset. Since we configured the execution of 10 epochs, we obtained 10 model snapshots for each fold. Therefore, for each fold, we obtained the predictions using the validation subsets in order to select the best epoch, and finally, for the best one, we obtained the predictions using the test subset, which the model had never seen. We used these test predictions to calculate the values for each one of the selected metrics, for each fold. Finally, to complete the whole process, we calculated the mean and the standard deviation, for each metric. The evaluation of the experiments published in this paper, as well as the hyperparameter optimization process, was carried out using the following evaluation metrics: accuracy, precision, recall and F-score. It is worth noting that we calculated the weighted metrics since the dataset is imbalanced, in terms of rows of data for each type of attack. \begin{table}
\begin{tabular}{c c c c c c} \hline \hline
**Max. Input Tokens** & **Accuracy** & **Precision** & **Recall** & **F-score** & **Training Time** \\ \hline
50 & 99.75\% & 99.75\% & 99.75\% & 99.75\% & 2:22:33 \\
100 & 99.84\% & 99.84\% & 99.84\% & 99.84\% & 2:45:20 \\
**150** & **99.84\%** & **99.85\%** & **99.84\%** & **99.84\%** & **3:15:47** \\
200 & 99.84\% & 99.85\% & 99.84\% & 99.84\% & 3:43:27 \\
250 & 99.84\% & 99.85\% & 99.84\% & 99.84\% & 4:20:27 \\
350 & 99.84\% & 99.85\% & 99.84\% & 99.84\% & 5:28:13 \\
500 & 99.84\% & 99.85\% & 99.84\% & 99.84\% & 7:42:27 \\ \hline \hline \end{tabular}
\end{table} 
Table 7: Metrics for the hyperparameter optimization step. In addition, these values allow us to compare our results with those reported in other publications. Considering the aforementioned information, we obtained 99.84%, for all the selected metrics. The calculated metrics may not provide conclusive information under certain conditions. However, these metrics may conceal some information that could be interpreted in a confusion matrix. Table 8 shows the confusion matrix for the first fold. The results are highly optimistic since the higher values are in the main diagonal and the remaining values are 0 or close to 0. In addition, two interesting conclusions were extracted from this matrix. The first is the number of false negatives that appear when the model predicted Infiltration, label number 5, where the right answer is Benign, label number 0. This means that, for the model, both classes are confusing for a minimum proportion of data. The other interesting finding is that the model exhibited lower recall and precision for certain minority classes within the datasets. For instance, SQL Injection attacks, representing an extremely low proportion of the total instances (only 17 samples), achieved a detection accuracy of merely 23.53%. Similarly, Brute-Force-Web attacks, with 122 samples, showed an accuracy of 86.89%, and Brute-Force-XSS attacks, with 46 samples, reached 91.3%.