## 3 Method
### Pre-Trained Model for the Second-Class Entities
#### 3.2.1. BERT Pre-Trained Language Model  
Pre-trained language models have advanced quickly in recent years. Pre-trained models, which primarily have the following two qualities, are crucial for many natural language processing (NLP) tasks: (1) It can be trained using a huge unlabeled text corpus. (2) No particular network structure needs to be created in order to use it for a variety of downstream NLP jobs that are not specific. Selecting one of several predetermined network architectures for fine adjustment can yield good results. According to Figure 5, the unit sum of three embedded features makes up the coding vector input by the BERT layer (length 512). Token embedding is used to represent words as vectors. The CLS flag, which can be used for later downstream NLP operations, is the first word. Segment embedding is a sentence vector that is used in classification tasks to distinguish between two sentences. Position embeddings serve to represent the position vectors that the Bert model has learned.  
Bidirectional encoder representation from transformers' (Bert) network architecture adopts a multi-layer transformer structure. It uses an attention mechanism instead of conventional recurrent neural networks (RNNs) and convolutional neural networks (CNNs) to convert the distance between two words at any position into one. This effectively handles  
Figure 4: NER model based on BERT-Bi-LSTM-CRF.  
Figure 5: Input representation for the Bert model.  
the challenge of long-term dependence in the process of feature extraction. The BERT network structure is shown in Figure 6, where \(\mathbf{S_{0}}\), \(\mathbf{S_{1}}\),..., \(\mathbf{S_{n}}\) are the input vectors of the model; \(\mathbf{W_{0}}\), \(\mathbf{W_{1}}\),..., \(\mathbf{W_{n}}\) are output vectors of the model; and \(T_{rm}\) is the transformer module.  
Bert is a multi-task model including two self-supervised tasks: masked language model (MLM) and next sentence prediction (NSP).  
A masked language model (MLM) describes the process of masking some words from the input corpus during the training stage, for example, \"refined benzene tanker side [mask]\", and then predicting the masked words based on context. In the experiment of BERT, 15% of the words in training samples were randomly masked. Sentences will be fed into the model repeatedly in order to learn model parameters. However, these words will not always be hidden. Specifically, 80% of the words will be replaced with masked characters, 10% will retain the original characters, and the rest 10% will be replaced with any other words. It is because that if a character in a sentence is 100% masked, some unregistered words will appear when tuning the model. The reason for adding random characters is that the transformer structure maintains a distributed representation of each input character.  
The purpose of the next sentence prediction (NSP) is to determine whether sentence B is the following of sentence A. If sentence B is the following of sentence A, it outputs \"is next\", otherwise it outputs \"not next\". Taking \"[CLS] refined benzene tank car rollover [SEP] no one is trapped\" for example, the output result of NSP is \"is next\". While \"[CLS] refined benzene tank car rollover [SEP] actual load of sand is 32 tons\", the output result is \"not next\". Two consecutive sentences chosen at random from the parallel corpus serve as the training data. The two sentences that were extracted from them have a 50% chance of remaining, proving that they are related contextually. Another 50% chance suggests that sentence B was chosen at random from the corpus and does not relate to sentence A contextually.