# An ERNIE-Based Joint Model for Chinese Named Entity Recognition
## 3 Methods
### Erniie-Joint  
As shown in Figure 4, ERNIE-Joint is an improved model of ERNIE, and the core part of it is still ERNIE. Therefore, the input format of ERNIE-Joint must be exactly the same as that of ERNIE. The input of ERNIE-Joint constructed by summing the following four parts:  
* Token IDs: This number means the ID of each token based on the dictionary of ERNIE-Joint. Figure 3: The different masking strategies between BERT and ERNIE. * Sentence IDs: ERNIE uses this number to determine which sentence the token belongs to. However, all the sentence ID numbers are \"0\" in this work because we input only one sentence at a time into ERNIE-Joint, not a sentence pair. * Position IDs: The Transformer cannot obtain position information. Therefore ERNIE-Joint uses position IDs to obtain the order of the tokens. * Segmentation IDs: This number indicates whether a character belongs to an entity or a phrase. Specifically, \"0\" means it belongs to the beginning of a Chinese entity or phrase, and \"1\" means it does not belong to the beginning. As mentioned before, the input of ERNIE-Joint is identical with that of ERNIE. Therefore, it is necessary to insert the special token [CLS] as the first token of a sequence and [SEP] as the final token. Given an input token sequence \(\mathbf{t}=\{t_{1},\dots,t_{D}\}\), the output of ERNIE-Joint is \(\mathbf{h}=\{h_{C},h_{1},\dots,h_{D}\}\) where \(h_{C}\) is the representation of [CLS], and the \(h_{d}\) corresponds to \(t_{d}\), being \(d=1,\dots,D\). For the NER task, the token-level categories probabilities for the token \(t_{d}\) can be obtained through:  
\[P_{d}=softmax(\mathbf{W}_{s}h_{d}+\mathbf{b}_{s}), \tag{1}\]  
where \(\mathbf{W}_{s}\in\mathbb{R}^{N*H}\) and \(\mathbf{b}_{s}\in\mathbb{R}^{N}\), that is, the token-level classifier matrix and bias. \(H\) is the dimension of final hidden state and \(N\) is the number of token-level categories. The category for token \(t_{d}\) can be obtained by:  
\[y_{d}=argmax(P_{d}). \tag{2}\]  
Figure 4: The architecture of ERNIE-Joint. The loss function for one token is:  
\[loss=\sum_{n=1}^{N}p(y_{d}^{n})\log[q(y_{d}^{n})], \tag{3}\]  
where \(p(y_{d}^{n})\) denotes the probability distribution of correct labels and \(q(y_{d}^{n})\) denotes the probability distribution of predicted labels. For the text classification task, the sentence-level category probabilities for token [CLS] can be obtained through:  
\[P_{c}=softmax(\textbf{W}_{c}h_{c}+\textbf{b}_{c}), \tag{4}\]  
where \(\textbf{W}_{c}\in\mathbb{R}^{M*H}\) and \(\textbf{b}_{c}\in\mathbb{R}^{M}\), that is, the sentence-level classifier matrix and bias. \(M\) is the number of sentence-level categories. The category can be obtained by:  
\[y_{c}=argmax(P_{c}). \tag{5}\]  
The loss function for the text classification task is:  
\[loss=\sum_{m=1}^{M}p(y_{c}^{m})\log[q(y_{c}^{m})], \tag{6}\]  
where \(p(y_{c}^{m})\) denotes the probability distribution of correct labels and \(q(y_{c}^{m})\) denotes the probability distribution of predicted labels. In order to fine-tune the ERNIE-Joint model by joint training both the two tasks, we define a unique cost function. Giving the input sentence \(\boldsymbol{x}\), the learning objective for joint training is to maximize the following conditional probability:  
\[p(\boldsymbol{y_{i}},\boldsymbol{y_{s}}|\boldsymbol{x})=p(y_{c}|\boldsymbol{ x})\prod_{d=1}^{D}p(y_{d}|\boldsymbol{x}), \tag{7}\]  
where \(\boldsymbol{y_{i}}\) denotes the correct number of the entities in the input sequence, and \(\boldsymbol{y_{s}}=\{y_{1},\dots,y_{D}\}\) denotes the correct sequence for the NER. Then, the optimization goal for joint training is to minimize this cost function:  
\[cost=\frac{1}{D+1}\{\sum_{d=1}^{D}\sum_{n=1}^{N}p(y_{d}^{n})\log[q(y_{d}^{n})] +\sum_{m=1}^{M}p(y_{c}^{m})\log[q(y_{c}^{m})]\}. \tag{8}\]  
Moreover, sentences are classified based on emotion or intent for common text classification tasks. However, additional annotations are necessary if we use a raw NER dataset for joint training based on the indicators above.