# Chinese Named Entity Recognition of Geological News Based on BERT Model
## 2 Materials and Methods
### BiLSTM-CRF  
Proposed by [PERSON] [16] et al. and based on the LSTM-CRF model, the BiLSTM-CRF model is a deep learning model that integrates feature engineering and serialization. The model structure is shown in Figure 2. It is mainly divided into a three-layer structure of a word vector input layer, a BiLSTM layer, and a CRF layer. The experimental process can be divided into three steps. First, the input of the model is a sequence of word vectors. Second, the probability vector of the corresponding label of each word is output on the BiLSTM layer. Finally, the result is corrected through the CRF, and the label sequence with the highest probability is output. These three parts are explained in detail below. Figure 1: CBOW model (**left**) and skip-gram model (**right**). The first part is the word vector input layer, where the sentence is input. Assuming that the sentence is \(X\), \(X\) is composed of \(n\) words (\(X_{1},X_{2},\ldots,X_{n}\)), denoted as \(X=(X_{1},X_{2},\ldots,X_{n})\). Each word in the sentence is mapped into a word vector using the word embedding method, corresponding to \(w_{0},w_{1},w_{2},w_{3},\ w_{4}\), etc. In the figure, the matrix formed by these word vectors is used as the input of the next layer. The second part is the BiLSTM layer, which has two hidden layers: a forward LSTM layer and a backward LSTM layer. The input word vector goes through the two hidden layers, and the results of these hidden layers are spliced together to determine the state of the final hidden layer. Based on the state of the final hidden layer, a linear layer is connected to perform a mapping operation to map the hidden layer matrix from \(n\) dimensions to K dimensions, where \(n\) is the dimension of the matrix and K is the number of labels. Finally, the BiLSTM layer will output the resulting matrix, which is denoted as \(\text{P}=(p_{0},p_{1},\ldots,p_{n})\), where \(p_{i}\) is the vector composed of the emission scores of \(w_{i}\) corresponding to each label, e.g., \(p_{0}=[1.5,0.9,0.1,0.08,0.05]\). The matrix P serves as the input matrix of the CRF layer. The last part is the CRF layer, which adds constraints between the labels and reduces the number of invalid predicted labels. Although the Softmax function outputs the label with the maximum probability corresponding to the word, the output labels are independent of each other. This means that the sequence is prone to unreasonable situations, resulting in a decrease in the accuracy rate, such that the time and organization entity of the \"I\" label may become adjacent. However, we know that according to the \"BIO\" labeling method, the entity tag must begin with \"B\". Thus, the \"I\" label of the time and organization entity cannot be adjacent. After the word vector passes through the BiLSTM and CRF layers, the score of the final sequence consists of two parts: the emission score of the BiLSTM layer and the transfer score of the CRF layer, as shown in Equation (1). The \(X\) on the left side of the equal sign  
Figure 2: The structure of the BiLSTM-CRF model. represents the scoring sequence, and the first part on the right side of the equal sign \(P_{i,y_{i}}\) represents the emission score of the \(y_{i}\) label in the \(i\)th word vector. The second part \(A_{y_{i-1},y_{i}}\) represents the emission score from \(y_{i-1}\) label to the \(y_{i}\) label. \[score(X,y)=\sum\limits_{i=1}^{n}P_{i,y_{i}}+\sum\limits_{i=1}^{n+1}A_{y_{i-1}, y_{i}} \tag{1}\]  
After calculating the score for each possible sequence, it is normalized using Softmax. The result is shown in Equation (2), which \(Y_{(x)}\) represents all possible labeled sequences. \[p(y|X)=\frac{e^{score(X,y)}}{\sum\limits_{y\in Y}(x)^{score(X,y)}} \tag{2}\]  
In the model training process, the log-likelihood function is used to optimize the model. The result is shown in Equation (3).