## 2 Related Work  
At present, the common methods for named entity recognition include dictionary and rule-based methods, statistical machine-learning-based methods, and deep-learning-based methods. The dictionary and rule-based named entity recognition methods usually achieve good results in a specific domain, but are time-consuming, laborious and less portable. With the development of machine learning technology, the machine learning method represented by CRF [14] can effectively reduce the labor cost by combining dictionary and rules, but they still have drawbacks, such as inefficient feature extraction and slow training. In recent years, deep learning methods based on neural networks have achieved great success in fields such as computer vision and speech recognition, and good results have been achieved in the research of named entity recognition, a key fundamental task in natural language processing. [PERSON] et al. [15] proposed a named entity recognition model for BiLSTM-CRF power defect text based on selectable word vectors. The model can identify the category information of specialized named entities in the power domain from a large number of power defect texts, thus constructing a large amount of power defect text data and facilitating the management of text data. [PERSON] et al. [16] used an Att-BiLSTM-CRF model combined with a self-attention mechanism to achieve excellent results on clinical named entity recognition (CNER). [PERSON] et al. [17] proposed a dynamic attention mechanism-based approach by stitching the character vector of raw text information and the word vector of domain information to further enhance the model performance. [PERSON] et al. [18] proposed a bidirectional long and short-term memory method based on attention and conditional random field layer (Att-BiLSTM-CRF) for chemical-named entity recognition to achieve document-level chemical memory, the method obtains document-level global information through the attention mechanism, enforces marker consistency among multiple instances of the same token in the document, and achieves good performance on small feature engineering, which is widely cited by researchers. [PERSON] et al. [19] proposed a neural network method called Dic-Att-BiLSTM-CRF (DABLC), which uses an efficient exact string matching method to match entities with dictionaries and constructs a dictionary attention layer to enhance the model performance by combining the dictionary matching method with a document-level attention mechanism. In response to the complexity and lack of context of medical terminology on Twitter, [PERSON] et al. [20] used a bidirectional long-short-term memory model (BiLSTM) to learn a large amount of contextual information, and then used convolutional neural networks (CNN) to generate character-level features, and labeled information incorporated into the CRF model to detect health-related entities from Twitter messages. [PERSON] et al. [21] used the attention mechanism to improve the vector representation in BiLSTM, designed several different attention weight assignment methods and combined them to effectively prevent significant information loss during feature extraction, and finally, combining BiLSTM with the CRF layer can effectively solve the problem of strong label dependence in sequences. The self-attentive mechanisms utilized in the above studies have few parameters and low complexity. Although they enhance the computational efficiency, they all suffer from the inability to learn the sequential relationships within the sequence. Furthermore, most of the embedding layers utilize the Word2 Vec model, resulting in context-independent shallow feature vectors that are insufficient for characterizing the multiple meanings of words. Pre-trained language models using deep learning not only overcome the limitations of machine learning methods, but also address the shortcomings of self-attentive mechanisms. [PERSON] et al.