## 4 Experiments and Results
### Experimental Results
#### 4.3.2 Performance Analysis of the Anomaly Detection Model  
To evaluate the effectiveness of the proposed industrial control device anomaly detection model based on the fine-tuned Llama3 model, this experiment compared the changes in the Accuracy metric before and after fine-tuning using germa-7b, llama-3-8b, and Phi-3-mini-4k as the foundational large language models. The performance differences of these three foundational models after fine-tuning for industrial control device anomaly detection were also analyzed. As shown in Figure 13, all three original models had zero capability for industrial control device anomaly detection and could not complete the task. However, the ability of the three models to detect anomalies in industrial control devices varied significantly after fine-tuning. The Phi-3-mini-4k model exhibited the poorest performance after fine-tuning, with an accuracy of only 68.4%. The germa-7b model demonstrated better performance, achieving an accuracy of 88.0%. The llama-3-8b model had the best anomaly detection capability, with an accuracy of 99.8%. Figure 14 further illustrates that the fine-tuned lla-ma-3-8b model, which showed the strongest performance in industrial control device anomaly detection, maintained excellent Accuracy across various types of anomalous data. The performance improvement is mainly attributed to the unique characteristics of industrial control device anomaly detection data. The original foundational large language models could not determine the anomaly status of industrial control devices based on the input. After fine-tuning the models in the same way, the ability to learn features of anomalous industrial control devices varied significantly due to the different foundational  
Figure 11: Accuracy comparison of different types of DFs. Figure 12: F1-Score comparison of different types of DFs. parameters of each model. Consequently, the anomaly detection capabilities of different models for industrial control devices showed substantial differences. In conclusion, selecting an appropriate foundational large language model is the first step in ensuring that the fine-tuned model can effectively perform anomaly detection tasks at a high level. During the fine-tuning process, if the number of training epochs is too small, the model cannot sufficiently learn the features of the samples, resulting in poor performance in identifying anomalies in industrial control devices. As the number of training epochs increases, the model learns the sample features more thoroughly, improving its ability to identify anomalous devices. However, once the number of training epochs exceeds a certain value, the model may overfit the sample features, including noise and outliers, leading to poorer performance on unseen new data and a significant decline in prediction accuracy. As shown in Figure 15, using the llama-3-8b-bnb-4 bit model with the optimal number of training epochs, changing the algorithm from linear to cosine increased the model's detection accuracy from 99.7387% to 99.8258%. With the linear algorithm, the fine-tuned model already achieved 100% accuracy in identifying anomalous devices for Allen-Bradley and Honeypot datasets, leaving no room for improvement. After switching to the cosine algorithm, the detection accuracy for devices in the Others dataset increased from 99.3671%  
Figure 14: Accuracy comparison of different types of DFs. Figure 13: Comparison of Accuracy after fine-tuning different models. to 99.5781%. This improvement is due to the cosine algorithm's ability to enhance the model's generalization capability, ultimately boosting the model's overall accuracy. Finding the appropriate number of training epochs often requires multiple experiments, which is very time-consuming and labor-intensive. Based on Equations (2)-(8), this chapter uses a sample size of 1200-5200 for training data and a sample size of 6039 for testing data.