# KGNER: Improving Chinese Named Entity Recognition by BERT Infused with the Knowledge Graph
## 3 Model
### Embedding Layer  
In the embedding layer, the BERT encoder can encode sentence trees as embedding representation. The embedding representation of BERT consists of token embedding, position embedding, and segment embedding. There is one significant difference between our model input and the general BERT. Our model input is a sentence tree instead of a token sequence. Therefore, it is crucial for BERT to convert the sentence tree into a sequence while preserving the original information of the sentences. **Token embedding**: For this work, our token embedding is in line with BERT. We adopt the vocabulary provided by Google BERT in this paper. Each token in the sentence tree is turned into an embedding vector of dimension \(H\) with the help of a trainable lookup table. Moreover, \([CLS]\) is considered as as a category tag and \([MASK]\) is regarded as a masked token in BERT. Nevertheless, tokens in the sentence tree are expected to be rearranged before the embedding operation is performed. **Seg embedding**: When multiple sentences are entered, BERT uses segmentation embedding to distinguish between the different sentences. In this work, we only use a sentence. \([A,A,A,A,...,A]\), a sequence of segment tags, is adopted to mark a sentence. **Position embedding**: For BERT, the position embedding contains all the structural information of the BERT's input sentence. Without position embedding, this will be treated as a bag-of-word model, resulting in a lack of structural information (i.e., the order of tokens). We are allowed to add the missing structured information to the unreadable rearranged sentence. \[E=K\text{-}Query(S,\mathbb{K}) \tag{2}\]  
Figure 2: Structure of the sentence tree. **Mask-Self-Attention**: The risk involved in using knowledge is that the original sentence can suffer from changes in meaning. To avoid affecting the meaning of the original sentences, some measures need to be taken. First, a sentence tree is constructed by knowledge, and each word in the sentence tree is encoded by absolute position. Then, the tokens in the sentence tree are flattened into a sequence of the token by means of their absolute-position index. In other words, the original sentence is followed by tokens in the branch. As shown in Figure 3, the sentence tree is rearranged as \"Jack is in Beijing Zhongshan park now capital China is_a city locate_in Tian'an'men west\". The advantage of this is that the original semantic information of the sentence is preserved, but the sentence is still unreadable. For example, there is no connection between [_now_] and [_capital_]. Fortunately, this can be solved by limiting the visible area of each token by the mask matrix. We introduce the visible mask matrix in Figure 4. The sentences of different branches are invisible to each other. Here is an example to explain this. [_China_] and [_city_] are not visible to each other; however, [_Beijing_], [_capital_] and [_China_] are visible each other. In this way, the words in the sentence not only acquire the corresponding knowledge information but also prevent the influence of irrelevant words. The visible matrix is defined as (3). \[M_{ij}=\left\{ \begin{array}{cc}0&w_{i}\odot w_{j}\\ -\infty&w_{i}\odot w_{j}\end{array} \right. \tag{3}\]  
where \(w_{i}\odot w_{j}\) indicates that \(w_{i}\) and \(w_{j}\) are in the same branch, while \(w_{i}\odot w_{j}\) are not. \(i\) and \(j\) are the absolute position index. To avoid the semantic changes caused by taking advantage of the sentence structure information in \(M\), we utilize a mask-self-attention, which is an extension of self-attention. Formally, the mask-self-attention is defined as (4). \[Q^{i+1},K^{i+1},V^{i+1}=h^{i}W_{q},h^{i}W_{k},h^{i}W_{v}, \tag{4}\]  
\[S^{i+1}=\text{softmax}\Bigg{(}\frac{Q^{i+1}K^{i+1^{+}}+M}{\sqrt{d_{k}}}\Bigg{)}, \tag{5}\]  
\[h^{i+1}=S^{i+1}V^{i+1} \tag{6}\]  
where \(W_{q}\), \(W_{k}\) and \(W_{v}\) are trainable model parameters. \(h^{i}\) is the hidden state of the \(i\)-th mask-self-attention blocks. \(d_{k}\) is the scaling factor.