# Military Equipment Entity Extraction Based on Large Language Model
###### Abstract  
The technology of military equipment entity extraction, a crucial component in constructing military knowledge bases, holds significant research value and theoretical importance for guiding the development and improvement of equipment support forces. In the military domain, equipment entities exhibit a phenomenon of nesting, where one entity is contained within another, and abbreviations or codes are frequently used to represent these entities. To address this complexity, this paper proposes a method named CoTNER for extracting entities. Initially, a large-scale language model is used to perform data augmentation with chain-of-thought on the original dataset, providing additional semantic and contextual information. Subsequently, the augmented dataset is fine-tuned on a small-scale language model to adapt it to the task of military equipment entity extraction and to enhance its ability to learn complex rules specific to the domain of military equipment. Additionally, a high-quality data filtering strategy based on instruction-following difficulty scoring is proposed to address the catastrophic forgetting issue that may occur during the fine-tuning of large language models. The experimental results demonstrate that the proposed military equipment entity extraction method outperforms mainstream traditional deep learning methods, validating the effectiveness of CoTNER. military equipment entity extraction; data augmentation with chain of thought; fine-tuning; high-quality data filtering +
Footnote †: journal:  
to distinguish between them with only a few prompt examples. Additionally, nested entities are present in intelligence texts within the military equipment domain. For example, in the text \"using a solid rocket engine helps it become a precise, reliable, and effective ballistic missile system\", the phrase \"ballistic missile system\" should be annotated as a \"system\" entity, but the model might mistakenly annotate the substring \"ballistic missile\" as a \"missile\" entity. Currently, LLMs can be categorized based on their parameter size into large-scale LLMs with hundreds of billions of parameters (e.g., GPT-3 175B [4]) and small-scale LLMs with tens of billions of parameters (e.g., Llama-2 7B [5]). Large-scale LLMs, with their vast number of parameters, can capture more complex and nuanced semantic information, thus performing better on complex tasks. However, they require more computational resources for training and inference, leading to higher deployment and usage costs in practical applications. In contrast, small-scale LLMs, though less performant, incur lower costs during fine-tuning, making them suitable for applications in the military domain where resources are limited and rapid response is required for equipment entity extraction tasks. To enhance the performance of LLMs in the entity extraction task within the military equipment domain, this paper proposes a method that leverages the synergy between large-scale and small-scale LLMs. The main contributions of this paper are as follows:  
* **Data augmentation with chain of thought:** Using GPT-3.5, the Chain of Thought (CoT) [6] method is employed to augment the original data, adding explanations for the entity label results for each piece of data. * **Fine-tuning with instructions:** The augmented data are then used to fine-tune Llama-3 8B with instructions, thereby improving the model's logical reasoning capabilities and its ability to learn complex rules in the military equipment domain. * **High-quality data filtering:** A high-quality data filtering strategy is proposed to address the issue of catastrophic forgetting. This strategy introduces an Instruction-Following Difficulty (IFD) score, which measures the complexity of entities in the text. Only data with IFD scores above a predefined threshold are selected for model fine-tuning, maximizing the retention of knowledge acquired by the base Llama-3 model during pre-training.