Janko at SemEval-2023 Task 2: Bidirectional LSTM Model Based on Pre-training for Chinese Named Entity Recognition
Year: 2023

Abstract:
This paper describes the method we submitted as the Janko team in the SemEval-2023 Task 2, Multilingual Complex Named Entity Recognition (MultiCoNER 2). We only participated in the Chinese track. In this paper, we implement the BERT-BiLSTM-RDrop model. We use the fine-tuned BERT models, take the output of BERT as the input of the BiLSTM network, and finally use R-Drop technology to optimize the loss function. Our submission achieved a macro-averaged F1 score of 0.579 on the test set.

heading: 1 Introduction
text: Named entity recognition (NER) is to identify entities with specific meanings from a given text. Named entities generally refer to entities with a specific meaning or strong referential meaning in the text, usually including personal names, place names, organizational structure names, etc. NER is a key task in natural language processing, and it is very useful in NLP tasks such as information extraction (Toda and Kataoka, 2005), question answering system (Mollá et al., 2006), and machine translation (Babych and Hartley, 2003). However, due to the nature of the language, NER is also a challenging problem. First, a word often has multiple meanings, leading to different categories of entities in different sentences. For example, Washington can mean either a place name or a person’s name. Secondly, the boundary of the named entity is vague, and its forms are various. It can be a simple noun, a gerund phrase, or even a complete sentence.
MultiCoNER 2 (Fetahu et al., 2023b) aims to identify named entities with finer precision, and the organizers divided the whole competition into 13 tracks based on language. Each data set (Fetahu et al., 2023a) is composed of a training set, a validation set, and a test set. The training and validation set provides labels for named entities,
and the test set provides text only. Compared with MultiCoNER (Malmasi et al., 2022b), the number of entity tags has been increased from 6 (Malmasi et al., 2022a) to more than 30, while the semantics are vaguer and the text length is shorter. In addition, mock errors are added to the test set to make the task more realistic and difficult. Similar to other NER tasks, MultiCoNER 2 uses B-I-O tags to label the tokens, where “B” indicates the first token of a concept, “I” indicates tokens inside of a concept, and “O” indicates tokens that don’t belong to any entity.
This paper explains our submission to the Chinese track in MultiCoNER 2. We implemented a simple neural network system based on a pretrained language model. Our submission ranked 14th among 21 teams with 0.579 F1 score in the test set. This paper is organized as follows: In Section 2, we will briefly introduce the research status of the NER task in Chinese. Section 3 describes the models and methods used. Section 4 introduces the implementation details and experimental results. Finally, this paper is summarized in Section 5.

heading: 2 Related Work
text: English NER task is based on characters, but the Chinese NER task is more difficult than the English NER task because the meaning of words is much larger than a single word and Chinese has no space and the basic unit is the word(Geng et al., 2022). The development of NER in Chinese can be roughly divided into three categories: rule-based approach, traditional machine learning approach, and deep learning approach.

heading: 2.1 Method based on rules
text: In a rule-based NER system, rules can be artificially designed based on domain-specific gazetteers and syntactic patterns (Krupke and Hausman, 1998; Humphreys et al., 1995; Mikheev et al., 1998). This method performs well when the vocabulary is suf-
958
ficient. However, due to the characteristics of the Chinese language, the rule-based NER method is time-consuming and can not exhaust the rules and patterns in Chinese entities. In practical application, the recall rate is low and the effect is poor. Moreover, a specific system can only be applied to a specific domain and has no generalization ability.

heading: 2.2 Method based on traditional machine learning
text: Under this category, there are two methods: unsupervised learning and supervised learning. A typical approach to unsupervised learning is timeclustering, a cluster-based NER system (Collins and Singer, 1999; Nadeau et al., 2006) obtained different clusters according to text similarity, and extracted related entities by representing different entity category groups through clusters.
Supervised learning converts NER into a multiclass classification task or a sequence labeling task. Given annotated data samples, machine learning algorithms are used to learn the features of training samples, and to identify similar patterns from unknown data. Common NER models of supervised learning include HMM (Bikel et al., 1999), C4.5 decision Tree (Sekine et al., 1998), MEM (Borthwick, 1999) and CRF (Lafferty et al., 2002).

heading: 2.3 Method based on deep learning
text: Deep learning has attracted a lot of attention for its success in various fields, and quite a few studies have applied deep learning to NER over the past few years (Guan and Liu, 2021; Mai and Zhou, 2022; Zhu and Wang, 2022; Ma et al., 2022). Compared with traditional linear models, deep learningbased models can learn complex and potential features from data through nonlinear activation functions. Therefore, it has become a trend to use deep learning models such as BiLSTM (Wei et al., 2016) and BERT (Devlin et al., 2018) to deal with NER problems in Chinese and remarkable results have been achieved.
For example, Huang et al. (2015) for the first time, applied the BiLSTM-CRF model to the NER task. In this model, first of all, the word is mapped to a word vector, and the scoring probability of each word corresponding to each label is obtained by learning context information from the BiLSTM layer. By learning the sequence-dependent information between tags, the final prediction results are obtained. Lattice-LSTM (Zhang and Yang, 2018) model encodes all words matched by a single char-
acter in a sentence as directed acyclic graphs. There is no word segmentation error in this model, and the Lattice-LSTM model has achieved good results on each data set. However, Lattice-LSTM may encounter conflicts when using external dictionaries. LR-CNN (Gui et al., 2019) model proposes to deal with such conflicts by using advanced semantics to narrow the weight of words. TENER (Yan et al., 2019) model improves location coding and self-attention in the transformer. By replacing absolute position coding with relative position coding and improving attention allocation, the transformerbased model can improve performance in the NER task and obtain excellent results. Li et al. (2020) proposed a FLAT model containing lexical information, which converted the Lattice structure into a set of fragments and incorporated trainable relative position coding.
Based on the above results, we first use BERT pre-training model to obtain word embeddings. To better obtain the scoring probability of the word corresponding to each label, we used bidirectional LSTM to obtain the context information. Finally, we used the R-Drop technique to compensate for the inconsistency of Dropout during training and testing.

heading: 3 Methodology
text: In this section, we will introduce the system architecture we use in the Chinese NER task. The system will be composed of three parts. The model we will eventually use to predict the test set files is called BERT-BiLSTM-RDrop.The architecture of the BERT-BiLSTM-RDrop model is shown in Figure 1.

heading: 3.1 BERT Embeddings
text: Transformer is an encoder-decoder architecture. BERT is a bidirectional encoder based on the transformer that uses an attention mechanism to extract information from the surrounding context. Models based on transformer architecture (Vaswani et al., 2017) have achieved very good performance in NER tasks. The Bert-based model uses only one of the encoders and is pre-trained using masking language modeling (MLM) (Taylor, 1953) and next sentence prediction (NSP) tasks. In this task, we treat the last layer of BERT as an embedding of text. In this task, we treat the last layer of BERT as an embedding of text.

heading: 3.2 BiLSTM Neural Network
text: LSTM (Hochreiter and Schmidhuber, 1997) is a kind of RNN, which is very suitable for modeling sequential data, but it cannot encode information from back to front. BiLSTM is a combination of forward LSTM and backward LSTM, with one forward processing input sequence and the other reverse processing sequence. After the processing is completed, the output of the two LSTMs is joined together. BiLSTM does a better job of capturing bidirectional semantic dependencies when classifying at a finer level, especially when the output at the current moment is not only related to previous states but may also be related to future states.

heading: 3.3 R-Drop
text: Dropout (Srivastava et al., 2014) is a regularization technology widely used for deep learning. It randomly ignores or blocks some neurons in a certain proportion during training so that the model is different each time the prediction is made, which can solve the problem of overfitting. R-Drop (Liang et al., 2021) strengthens the robustness of the model to Dropout by adding a regularization term. R-Drop
acts on the output layer of the model to make up for the inconsistency of Dropout during training and testing and can be widely applied to training different types of deep models. During training, R-Drop will make the same input Dropout twice. Due to the characteristic of Dropout, two different output distributions will be obtained. R-Drop operation attempts to minimize the Kullback-Leibler (KL) divergence between these two output distributions to regularize model prediction, which is:
LiKL = 1
2 (DKL (Pw1 (yi | xi) ∥Pw2 (yi | xi)) +DKL (Pw2 (yi | xi) (1)
The loss function from the two Dropout is:
LiNLL = − logPw1 (yi | xi)− logPw2 (yi | xi) (2)
The final loss function is the sum of the two:
Li = LiNLL + α · LiKL (3)

heading: 4 Results
text: We used the Pytorch and Huggingface libraries to implement all the models in our experiment. Experiments were performed on a machine equipped with Intel® i7-12700F CPU, Nvidia® 3080 GPU, and 32 GB of RAM running Windows 10. The training time of the BERT-BiLSTM-RDrop model is about 4 hours. We used BERT-base-chinese as the training model. The results are evaluated by F1 scores:
Precision = TP
TP + FP (4)
Recall = TP
TP + FN (5)
F1 = 2 ∗ Precision ∗Recall Precision+Recall
(6)
During the training process, the BERT-BiLSTMRDrop model trains the data set provided by the organizer and saves the parameters of the 50th epoch for the subsequent test set prediction. The results are shown in Table 1. All results are final submissions to the system.
An ablation experiment was performed as shown in Table 1. It can be seen that BiLSTM can bring a large improvement because BiLSTM allows the network to obtain both forward and backward information about character sequences at each time step. Limited improvement from RDrop is expected since RDrop only reinforces the robustness of the model to Dropout by making up for inconsistencies in training and testing.
The initial learning rate was set to 1e-5 with a batch size of 4 and the α hyperparameter was set to 1. The hyperparameters used are shown in Table 2.
We tried some popular models in the verification set, and the results are shown in Table 3. It can be seen that the BERT-BiLSTM-RDrop has a better effect. We also tried other Bert-based models, such as BERT-base-uncased, RoBERTa-base, and XLMRoBERTa-base, and surprisingly, the results were not as good as BERT-base-chinese. In addition, we tried to use the integrated network of BiLSTM and CNN to replace the single BiLSTM and average method of voting method selection, but it is a pity that there is no significant improvement in measurement.
The best test set prediction submitted by our team was generated by the BERT-BiLSTM-RDrop model. Finally, Janko team ranked 14 out of 21 teams with a 0.579 F1 score.

heading: 5 Conclusion
text: In this paper, we described our method, which is based on the pre-trained BERT model and is used
to predict the NER task SemEval 2023 Task 2: Multilingual Complex Named Entity Recognition. We implemented the BERT-BiLSTM-RDrop model and prove the validity of the different model compositions. Our submission achieved a macro-averaged F1 score of 0.579 on the test set. In the future, we will continue to improve the system, use more advanced models, and try model integration methods.

heading: Future Work
text: In the future, we will continue to improve the system, use more advanced models, and try model integration methods