## 3 Overview of the Model  
### Named Entity Recognition Model Based on the Combination of Characters, Words, and Pinyin  
The model employed in this paper utilizes character-level input, which means that some features are influenced by ambiguous boundaries. To address this issue, the paper uses pre-trained word embeddings to augment the accuracy of named entity recognition. By introducing word embeddings, the model can better capture semantic relationships between words and comprehend words in context. The Chinese named entity recognition model is based on the LEBERT model. It integrates semantic information of over 12 million Chinese words pre-trained on a large-scale, high-quality dataset [19]. Additionally, it utilizes unique pinyin vectors of Chinese characters. It uses a BiLSTM to make predictions on character sequence labels and finally optimizes the predicted labels using CRF. The model demonstrates better performance compared to the original model on multiple Chinese named entity recognition datasets, showing significant improvements in recognition rates for diverse categories and heightened precision for each label. The model is illustrated in Figure 1.  
Figure 1: Named entity recognition model based on the combination of characters, words, and pinyin. In the embedding of pinyin representation, this paper adopts a similar approach to the generation of pinyin embeddings as ChineseBERT. Firstly, the open-source pygpinyin package is used to obtain the pinyin sequence for each character. Then, for each character's pinyin, completion is applied. A CNN model with a width of 2 is utilized to process the sequence, and the resulting sequence embedding is derived through max-pooling. This guarantees that the output dimension is not affected by the input sequence length. The input sequence length is fixed at 8. When the length of the sequence is less than 8, the remaining slots are filled with the special character \"-\". This is illustrated in Figure 2.  
For the given input text, it undergoes the initial processing step where it is fed into the embedding layer of BERT. This layer generates token embeddings, position embeddings, and segment embeddings for the sentence. Subsequently, the sentence is fed into the initial transformer layer of BERT, which yields the partial word features \(h_{1},h_{2},\ldots h_{n}\). Next, the word set embedding vector \(w_{i}=\{w_{i1},w_{i2},\ldots w_{im}\}\) is obtained, where \(w_{i}\) represents the word embedding vector set for the \(i\)-th character. The words embedding \(w_{i1}\), \(w_{i2}\), and so on are derived from the pre-trained word embedding lookup table of Tencent AI Lab Embedding Corpus. Additionally, there is a discrepancy between the dimensions of the word embeddings in the lookup table and those of the model. To ensure the alignment between the word vectors \(w_{ij}\) and the character vectors \(h_{i}\), a non-linear transformation of the word vectors is required:  
\[x_{ij}=W_{2}\big{(}tanh\big{(}W_{1}w_{ij}+b_{1}\big{)}\big{)}+b_{2}\]  
where \(W_{1}\) is a \(d_{c}\)_by_\(d_{w}\) matrix, \(W_{2}\) is a \(d_{c}\)_by_\(d_{c}\) matrix, and \(b_{1}\) and \(b_{2}\) are scalar bias. \(\mathrm{d_{c}}\) represents the hidden size of BERT, and \(d_{w}\) represents the dimension of word embedding. The above parameters, together with the tanh function, implement the transformation of word embedding. In order to combine the lexical features and pinyin features with character information, inspired by LEBERT, we designed a novel information adapter (as shown in Figure 3). Figure 3: Information adapter. Figure 2: Generation of pinyin embedding. The information adapter uses attention mechanism to integrate character features and lexical features into BERT, and then adds pinyin information to the adapter. We input the pinyin embedding vector \(p_{i}\), the character embedding vector \(h_{i}\), and the word embedding vector of the word into the information adapter (IA) layer.