# Military Equipment Entity Extraction Based on Large Language Model
## 4 Experimental Results
### Ablation Experiments  
To further validate the contributions and impacts of different modules within the CoTNER method on entity extraction tasks in the military equipment domain, this study conducted ablation experiments. Specifically, it explored the effects of the fine-tuning with instructions module, the data augmentation with the chain of thought module, and the high-quality data filtering module on the model's performance.  
Table 7 presents the results of the model on the ordinary and nested entity extraction test sets. The models in the \"Fully supervised\" row utilize the instruction fine-tuning learning paradigm, whereas those in the \"Few-shot\" row do not employ the instruction fine-tuning paradigm. The experimental results are analyzed as follows:  
(1) For the few-shot learning paradigm, Llama3 + CoT achieves performance of 85.77 and 69.36 on the CoNLL2003 and ACE2005 datasets, respectively. In contrast, when using the fully supervised learning paradigm, Llama3 + CoT results improve to 92.88 and 87.79 on the CoNLL2003 and ACE2005 datasets, respectively. This indicates that instruction fine-tuning is crucial for entity extraction tasks, particularly evident on the more complex nested ACE2005 dataset. Through fine-tuning on annotated data, the model effectively learns the diversity of complex entities, context relationships, and features, including nested and discontinuous entities, thereby achieving more precise identification of complex entity boundaries.  
(2) After integrating the data augmentation with the chain of thought module into the base Llama3 model, Llama3 + CoT shows an increase in F1 score from 89.32 to 92.88 on the CoNLL2003 dataset and from 82.43 to 87.79 on the ACE2005 dataset. The experimental results demonstrate significant performance improvement by incorporating the data augmentation with the chain of thought module in prompts and training data, with more pronounced enhancements observed on nested datasets.  
(3) Based on the application of the fine-tuning with instructions module and the data augmentation with the chain of thought module, the model achieved F1 scores of 92.88 and 87.79 on the CoNLL2003 and ACE2005 datasets, respectively, without the high-quality data filtering module. However, after incorporating the high-quality data filtering module, the F1  
\begin{table}
\begin{tabular}{c c c c c c} \hline \hline  & **Method** & **CoNLL2003** & **OnteNotes5.0** & **ACE2004** & **ACE2005** \\ \hline \multirow{3}{*}{Fully supervised} & Llama3 & 89.32 & 90.24 & 82.27 & 82.43 \\  & Llama3 + CoT & 92.88 & 89.85 & 87.86 & 87.79 \\  & Llama3 + CoT + Data filtering & 93.12 & 91.24 & 88.31 & 88.03 \\ \hline Few-shot & Llama3 + CoT & 85.77 & 81.01 & 70.16 & 69.36 \\ \hline \hline \end{tabular}
\end{table}
Table 7: A comparison of F1 scores in ablation study results.  
scores increased to 93.12 and 88.03, respectively. These results demonstrate that the addition of the high-quality data filtering module can further enhance the model's performance.  
Additionally, subsets containing the top 5%, 10%, 15%, and 20% of training data based on IFD scores were constructed, and the base Llama3 model was fine-tuned using these subsets. Performance changes were analyzed and compared. Figure 4 illustrates the variation in model performance scores with increasing data volume, calculated as the ratio of the F1 scores obtained on the subset to those obtained on the entire dataset. Experimental results indicate that using the top 10% of high-quality data based on IFD scores allows the proposed model to surpass the performance achieved when trained on the entire dataset, demonstrating that the high-quality data filtering strategy effectively mitigates the issue of catastrophic forgetting.