# KGNER: Improving Chinese Named Entity Recognition by BERT Infused with the Knowledge Graph
## 2 Related Work
### Pre-Trained Language Models  
In terms of deep learning, in order to possess a deeper model, we need to feed it adequate labeled data. In the NLP domain, labeled data are an expensive resource. Pre-Trained Models (PTMs) that were pre-trained from a great deal of unlabeled data have led to prominent performance gains in many NLP tasks. Two major paradigms are summarized: shallow word embedding and pre-trained encoders. **Shallow word embedding**: This class of PTM paradigms is what we usually call \"word vectors\", whose main feature is learning context-independent static word embedding, which is mainly represented by Word2 vec (CBOW [22], Skip-Gram [22]), Glove [23], etc. The main drawback of shallow word embedding is that word embedding is context-independent, and the embedding vector of each word is always the same. Therefore, it cannot solve the problem of multiple word meanings. **Pre-trained encoders**: The second class of PTM paradigms is the pre-trained encoder, which overcomes the issue of multiple word meanings by yielding a vector of contextually relevant words. This class of pre-trained encoders output vectors is called \"context-sensitive word embedding\". The representatives of this class are ELMo, GPT, Bert, etc. To solve the problem of multiple-meaning words, [PERSON] et al. [14] proposed using ELMo to distinguish the semantics of words using the semantics of the context. A two-stage process of pre-training and feature extraction is used. However, ELMo employs a tandem bidirectional RNN to obtain the semantics of the context. GPT [24], proposed by the OpenAI team in 2018, has a similar idea to ELMo in that both use a two-stage feature extraction process. The difference is that GPT replaces the RNN in the second stage using a Transformer with a better feature extraction ability to better capture long-range linguistic structures, but uses a one-way language model that can only accept semantic information from the above context. Google proposed the BERT model in late 2018 [13], which uses precisely the same two-stage structure as GPT: firstly, language model pre-training; secondly, solving downstream tasks using a Fine-Tuning model. The BERT model mainly addresses the problem that the GPT model does not use a bidirectional language model in the pre-training stage. Unlike previous language representation models, BERT is aimed at pre-training deep bidirectional representations by jointly adjusting the left and proper contexts in all layers. The Transformer Encoder that was used internally can capture the semantic information of a textual context using the self-attention mechanism. The word vectors obtained using the BERT model have better semantic properties than language models such as Word2 vec. In generic domains, the \(F1\) value has reached a high score of 93.09 on the named-entity recognition task of CoNLL2003. [PERSON] et al. [25] added a softmax to BERT to achieve a state-of-the-art performance on Chinese word segmentation(CWS). Some scholars [26; 27] showed that the model using character features of BERT significantly outperformed the static embedding-based approach on Chinese NER and Chinese part of speech (POS) tags. The methods in Section 2.1 integrate discrete and structured knowledge by designing different neural network structures. [PERSON] et al. [28] proposed Lexicon-Enhanced BERT (LEBERT) for Chinese sequence labeling, which causes BERT layers to consume the external lexicon by means of a Lexicon Adapter layer. Compared with existing approaches, their model promotes the fuller integration of lexical knowledge at the shallow level of BERT. In the last two years, generative pre-training models have also been heavily studied in addition to the above two pre-training paradigms. [PERSON] et al. [29] proposed Bidirectional and Auto-Regressive Transformers (BART). [PERSON] et al. [30] proposed Unified Text-to-Text Transformer's T5. They absorbed the features of BERT's bidirectional encoder and GPT's left-to-right decoder and built on the standard sequence-to-sequence Transformer model.