# Military Equipment Entity Extraction Based on Large Language Model
## 4 Experimental Results
### Comparative Experimental Analysis  
In the comparative experiments on public datasets, traditional deep learning models such as BiLSTM+CRF [12] and BERT-MRC [13] were chosen as benchmark models due to their effective performance in entity extraction tasks. Additionally, several LLM-based entity extraction models were selected as baselines to evaluate the entity extraction effectiveness of the proposed method, including two few-shot learning models GPTNER [17] and PromptNER [18], and one supervised fine-tuning model UniNER [19]. Table 5 presents the entity extraction performance of different models on the public datasets. The experimental results indicate that among the traditional deep learning models mentioned, BERT-MRC exhibits the best performance in entity extraction. Due to the Transformer architecture's global attention mechanism, BERT effectively captures long-distance dependencies. Moreover, BERT-MRC addresses entity nesting issues in entity extraction tasks by transforming the task into a machine reading comprehension format. Among all LLM-based models, UniNER demonstrates the best performance. Its strength lies in leveraging the large-scale language model ChatGPT for data annotation, enabling the extraction of deep semantic knowledge that enhances entity recognition accuracy. However, UniNER, using the smaller-scale base model Llama2 for fine-tuning, shows relatively weaker logical inference capabilities. Its performance in recognizing complex entities like nested structures still requires improvement. Additionally, fine-tuning large language models with extensive data may lead to catastrophic forgetting issues, thereby reducing entity extraction performance. The entity extraction method proposed in this paper shows significant improvements over the best-performing baseline model, UniNER. Specifically, the F1 scores on the OntoNotes 5.0, ACE 2004, and ACE2005 datasets increased by 2.14, 0.81, and 0.43, respectively. These results demonstrate that incorporating chain of thought reasoning into training data using large-scale LLMs effectively enhances the model's logical reasoning capabilities. Furthermore, fine-tuning smaller LLMs with the augmented data further adapts the model to the entity extraction task, thereby improving entity extraction performance. In the comparison experiments on the military equipment dataset, the same six models were selected to validate the effectiveness of the CoTNER method in entity extraction tasks within the military equipment domain. Table 6 shows the entity extraction performance of all models on the self-built military equipment dataset. \begin{table}
\begin{tabular}{c c c c} \hline \hline
**Model** & **Precision** & **Recall** & **F1** \\ \hline BiLSTM+CRF & 79.77 & 81.77 & 80.76 \\ BERT-MRC & 82.58 & 83.42 & 83.02 \\ GPTNER & 83.29 & 85.11 & 84.20 \\ PromptNER & 84.36 & 83.44 & 83.91 \\ UniNER & 86.21 & 85.08 & 85.64 \\ CoTNER & 88.93 & 91.25 & 90.07 \\ \hline \hline \end{tabular}
\end{table} 
Table 6: A comparison of experimental results on military equipment datasets. \begin{table}
\begin{tabular}{c c c c c} \hline \hline
**Model** & **CoNLL2003** & **OnteNotes5.0** & **ACE2004** & **ACE2005** \\ \hline BiLSTM+CRF & 91.03 & 86.28 & - & - \\ BERT-MRC & 93.04 & 91.11 & 85.98 & 86.88 \\ GPTNER & 90.91 & 82.2 & 74.2 & 73.59 \\ PromptNER & 83.48 & - & - & - \\ UniNER & - & 89.1 & 87.5 & 87.6 \\ Ours & 93.12 & 91.24 & 88.31 & 88.03 \\ \hline \hline \end{tabular}
\end{table} 
Table 5: A comparison of F1 scores on public datasets. The experimental results indicate that, with only 3000 pieces of military equipment domain data, models based on large language models (LLMs) significantly outperform those based on traditional deep learning methods in entity extraction performance. This is because traditional deep learning methods start from scratch and rely on domain-specific datasets, making it challenging to achieve good performance with limited data.