# LLM-Based Cyberattack Detection Using Network Flow Statistics
## 2 Materials and Methods
### Training the Cyberattack Detection System
#### 2.3.5 Fine-Tuning Process  
Training consists of adjusting the weights of each neuron of an ANN, such that predictions can be adjusted as closely as possible to the expected outputs. Fine-tuning involves using a pre-trained model as a starting point to solve a different problem. In essence, we load a pre-trained model and train it with our dataset, to adjust its weights to solve a new type of problem. Thus, instead of starting with a random weight configuration, we start with a weight configuration focused on solving a certain type of problem. In our case, the T5 model was pre-trained with the C4 [26] dataset, Colossal Clean **C**rawled **C**orpus, which was composed of hundreds of gigabytes of English text obtained from the Internet. Subsequently we applied fine-tuning with another dataset, focused on an entirely different scope, like that selected for our experiments. \begin{table}
\begin{tabular}{c c c c c} \hline \hline \multirow{2}{*}{**Types of Attacks**} & \multicolumn{2}{c}{**After Cleaning**} & \multicolumn{2}{c}{**Undersampling 20\% Benign**} \\ \cline{2-5}  & **Rows** & **Prop.** & **Rows** & **Prop.** \\ \hline Benign & **13,352,392** & **85.02\%** & **2,670,478** & **53.16\%** \\ Malignant & **2,352,951** & **14.98\%** & **2,352,951** & **46.84\%** \\ \hline \hline \end{tabular}
\end{table} 
Table 6: Dataset statistics grouped by benign/malignant after applying undersampling. Transfer Learning could be another option. This technique is similar to fine-tuning but with a particular difference. Fine-tuning updates all the weights of an ANN, unlike Transfer Learning, in which several layers are locked, such that they retain their weights, and only the weights of the unlocked layers are adjusted. Therefore, we decided to fine-tune all the weights, as the purpose of the model is being radically altered. Once we had decided to fine-tune an existing snapshot of the T5 model, using the SimpleT5 [51] wrapper, the next step was to optimize the hyperparameters. SimpleT5 is a wrapper built over PyTorch 2.0.1, to perform fine-tuning of T5 without modifying its architecture. This package provides users with different functions to use different sizes of the T5 model by only developing a few lines of code. It contains functions for downloading pre-trained snapshots and assigning them to a new instance, training models, and predicting values or loading models previously trained with this package. SimpleT5 has several hyperparameters available for use, which we carefully analyzed to select the most appropriate values for some and identify others to be optimized. Once this task is completed, the fine-tuning of the model can be successfully executed. Therefore, SimpleT5, provides users with the following hyperparameters:  
* max_epochs is the maximum number of epochs for the training step. It is set to 10, since LLMs have a high level of maturity and require relatively few epochs to achieve a good model. Therefore, after performing the preliminary experiments, we observed that the best-performing model of an experiment was never selected beyond the tenth epoch. Depending on the experiment, an overfitting could appear in one of the first epochs, even in the second one. Hence, 10 is a sufficiently large value. * target_max_token_len contains the maximum number of output tokens. Since the selected model performs a classification task, it is unnecessary to set an excessively large value. It should be pointed out that the output of the model is a numerical value between 0 and 14. The maximum number of tokens is 512, which substantially exceeds the number typically required. Preliminary experiments concluded that the minimum number of tokens was 3 and we observed that the values for the selected metrics maintained the same trend for any value between 3 and 512. In addition, this parameter has the same behavior as the variable source_max_token_len parameter in terms of the relationship between the number of tokens and training time. Therefore, setting 3 as its value significantly reduces training times.