YNUNLP at SemEval-2023 Task 2:The Pseudo Twin Tower Pre-training Model for Chinese Named Entity Recognition
Year: 2023

Abstract:
This paper introduces our method of developing a system for SemEval 2023 Task 2: MultiCoNER II Multilingual Complex Named Entity Recognition, Track 9-Chinese. In this task, we need to identify entity boundaries and category labels for the six identified categories. The focus of this task is to detect fine-grained named entities whose data set has a fine-grained taxonomy of 36 NE classes, representing a realistic challenge for NER. We use BERT embedding to represent each character in the original sentence and train CRF-Rdrop to predict named entity categories using the data set provided by the organizer. Our best submission, with a macro average f1 score of 0.5657, ranked 15th out of 22 teams.

heading: 1 Introduction
text: NER (Named Entity Recognition) is a traditional NLP task designed to identify text fragments belonging to predefined categories in free text(Curran and Clark, 2003). NER is an important basic tool for many NLP tasks such as information extraction, question answering system, syntactic analysis and machine translation. Named entities generally refer to entities with specific meaning or strong reference in the text. Academically, it usually includes three categories: entity, time and number, and seven categories: personal name, place name, organization name, time, date, currency and percentage(Chen et al., 2022). NER extracts the above entities from the unstructured input text and can identify more categories of entities based on business requirements.
Chinese named entity recognition(Gui et al., 2019) plays an important role in the field of natural language processing. Compared with English, Chinese NER is more challenging. First of all, Chinese word boundaries are vague, and there are no delimiters, such as Spaces, to clarify word boundaries. If Chinese NER adopts character-level model, there
will be semantic loss and boundary information loss. On the other hand, if we use the word-level model, the wrong word segmentation can also degrade performance. There are also more complex properties in Chinese, such as complex combinations, entity nesting, indefinite length, and neologisms on the Internet. In addition, Chinese is not case-sensitive and root-affix, and lacks the expression of a lot of semantic information. Named entity recognition is very important because in many applications we must extract the entities in our vocabulary(Wang et al., 2022).
SemEval 2023 Task 2: MultiCoNER II Multilingual Complex Named Entity Recognition(Fetahu et al., 2023b). This task aims to solve the problem of fine-grained named entity recognition. The mission has 13 tracks, of which tracks 1-12 are monolingual, including English, Spanish, Hindi, Bangla, Chinese, Swedish, Farsi, French, Italian, Portuguese, Ukrainian and German(Malmasi et al., 2022b). Participants trained a model that only worked in one language. Track 13 is a multilingual track, where participants need to use data from 12 languages to train a single multilingual NER model for all languages, which should be able to process single-language data from any language. We took part in the Chinese monolingual program.

heading: 2 Related Work
text: The term named entity recognition first appeared in Message Understanding Conferences (MUC6), which mainly focuses on information extraction(Hirschberg and Manning, 2015). In addition to information extraction evaluation task, the MUC6 also introduced a new evaluation task namely named entity recognition task. When a task is first proposed, it defines only a few generic entity categories, such as places, agencies, people, and so on. At present, named entity recognition task has penetrated into various vertical fields, such as medical treatment, finance and so on. The main methods
1619
of named entity recognition include dictionary and rule based method, traditional machine learning method and deep learning method(Li et al., 2020).

heading: 2.1 Dictionaries and rules based methods
text: The rules-based NER system(Humphreys et al., 1998) relies on human-made rules. Rules can be designed based on domain-specific gazetteers and syntactic lexical patterns. Better-known systems include LaSIE-II, NetOwl, Facile, and SAR. A rulesbased system is a good choice when the vocabulary is exhaustive. However, high accuracy and low recall rates are often observed from such systems in specific areas due to their specific rules and incomplete vocabulary, and these systems cannot be transferred to other areas.

heading: 2.2 Traditional machine learning based methods
text: The typical method of unsupervised learning is clustering(Gong et al., 2003). Clustering based NER system extracts related entities through context similarity clustering. Collins et al. used only a small amount of seed tagging data and seven characteristics, including spelling, entity context, and entity itself, for entity recognition(Neelakantan and Collins, 2015). Nadeau et al. proposed an unsupervised system for the construction of a local name dictionary and the ambiguity resolution of named entities(Nadeau and Sekine, 2007). The system is based on a simple and efficient heuristic method that combines entity extraction and ambiguity elimination.
With supervised learning, NER can be translated into multiple classification or sequence labeling tasks. Given labeled data samples, carefully designed features can be used to represent each training example. The model is then learned using machine learning algorithms to identify similar patterns from unknown data. Many machine learning algorithms have been applied to supervised NER(Yang et al., 2018), including Hidden Markov Model (HMM), decision tree, maximum entropy model, support vector machine (SVM) and Conditional Random Fields (CRF).

heading: 2.3 Deep learning based methods
text: Word sequence-based model(Sboev et al., 2021): English and most other languages naturally divide words by Spaces. The early Chinese NER model also follows the English NER model of word segmentation before prediction. However, word seg-
mentation errors are unavoidable in the word segmentation stage, which will transfer the errors to the subsequent modules and affect the recognition ability of the model.
Model based on word sequence: In order to avoid word segmentation errors caused by model based on word sequence, Chinese NER model since 2003 has mostly carried out further feature extraction and sequence labeling at the level of word sequence(Mai and Zhou, 2022; Guan and Liu, 2021).
Model integrating external information(Tsai et al., 2022): The simple model based on word sequence only uses the semantic information of characters, which is obviously insufficient in the amount of information. Therefore, the subsequent research considers integrating all kinds of external information into the character sequence, such as peripheral information, pinyin information, dictionary information, etc. Dictionary information is the most widely used external information.

heading: 3 Methodology
text: In this part, the model adopted by us for Chinese program of SemEval-2023 Task 2 is introduced in detail. This model consists of three parts, which we call the BERT-CRF-Rdrop model. Figure 1 shows the architecture of the BERT-CRF-Rdrop model.

heading: 3.1 BERT
text: The full name of BERT’s model is: Bidirectional Encoder Representations from Transformer. As can be seen from the name, the goal of BERT(Devlin et al., 2018) model is to use large-scale unannotated corpus to train and obtain representations of text containing rich semantic information, namely, semantic Representation of text, and then fineadjust the semantic representation of text in a specific NLP task, and finally apply it to the NLP task. In NLP method based on deep neural network, the word/vocabulary in text is usually represented by one-dimensional vector (generally called "word vector"). On this basis, the neural network will take the one-dimensional word vector of each word or vocabulary in the text as input, and after a series of complex transformations, output a one-dimensional word vector as the semantic representation of the text. In particular, we usually hope that semantically similar words/vocabularies are close to each other in the feature vector space, so that the text vector converted from the word/vocabulary vector can con-
tain more accurate semantic information. Therefore, the main input of BERT model is the original word vector of each word/vocabulary in the text, which can be initialized randomly or pre-trained using algorithms such as Word2Vector(Jiang et al., 2018) to take the initial value. The output is the vector representation of each word/vocabulary in the text integrated with the semantic information of the full text.

heading: 3.2 CRF
text: The basic definition of CRF(Huang et al., 2015) is: Let X and Y be random variables and the conditional probability distribution of Y under given X conditions. If the random variable Y forms a Markov random field of an undirected graph, the conditional probability distribution is called CRF. Corresponding to Markov can be understood as, if random variable Y forms an undirected graph, and every variable Y in the graph satisfies Markov (at least satisfies one of global, local and paired Markov), then it is called CRF. Where X is the input variable, namely the observation sequence that needs to be marked, and Y is the output variable, representing the state or marker sequence. In the field of natural language processing, the common input variable X and output variable Y have the same graph structure.
p(y | x) = 1
Z(x)
T∏
t=1
exp    K∑
k=1
θkfk ( yt, yt−1,xt )    (1)
Generally speaking, CRF formulas have two components: 1. Normalization: The right side of the equation has no probability, but has values and characteristics. However, the expected output is a probability, so normalization is required. The normalization constant Z(X) is the sum of all possible sequences of states, making the total one. 2. Weights and characteristics: This part can be regarded as a Logistic regression formula with weights and corresponding characteristics. Maximum likelihood estimation is used for weight estimation, and features are defined by ourselves.

heading: 3.3 R-drop
text: R-drop(Zhuang and Zhang, 2022) is a regularization strategy proposed on the basis of Dropout. Its main idea is to make the fractional model generated by dropout have uniformly distributed outputs for the same input data. Specifically, for each training sample, R-Drop minimizes the KL-divergence
between the output distributions of the two submodels generated by dropout. Due to the randomness of the Dropout, it is possible to approximate the path network that the input X walks through twice as two slightly different models. Theoretical analysis shows that R-Drop can reduce the degree of freedom of model parameters, supplement the loss, thus reducing the complexity of model space and improving the generalization of the model. One of the loss functions is the regular cross entropy, training data is (xi, yi), the model is Pθ(y|x), the cross entropy of each sample is Li = - log Pθ(yi|xi), and in the case of "Dropout twice," we can think of the sample as having gone through two slightly different models, P (1)θ (yi|xi), P (2) θ (yi|xi), so one part of the loss function is:
L1 = − log P(1)θ (yi | xi)− log P (2) θ (yi | xi) (2)
The other part of the loss function is the KL divergence, in order to make the two outputs as consistent as possible.
L2 = 1
2 [KL(P
(1) θ (yi | xi) | P (2) θ (yi | xi))
+KL(P (2) θ (yi | xi) | P (1) θ (yi | xi))]
(3)
The final loss of Rdrop is the weighted sum of L1 and L2.
L = L1 + αL2 (4)

heading: 4 Dataset
text: The SemEval-2023 Task 2 asks participants to develop complex named entity recognition systems for 12 languages, and we have experimented with Chinese(Fetahu et al., 2023a).
In this task, we used official raw data to train and test our model. The data set for this semeval task consists mainly of three sources: Low-Context Wikipedia(Meng et al., 2021), MSMARCO Question(Bajaj et al., 2016)and ORCAS Search Query(Craswell et al., 2020).
Each line of text in the data set of SemEval-2023 Task 2 belongs to a sample of languages(Fetahu et al., 2023a): English, Spanish, Hindi, Bangla, Chinese, Swedish, Farsi, French, Italian, Portuguese, Ukrainian and German. It consists of 6 entity types: Location(LOC), Creative Work(CW), Group(GRP), Person (PER), Product (PROD), Medical (MED). Participants must use their systems to accurately detect entities and submit predictions of tasks(Malmasi et al., 2022a). Track 9 - Chinese provides 9,759 training data, 506 validation sets and at least 100,000 final test data.

heading: 5 Results
text: We use accuracy, recall rate and F1 to evaluate the performance of our proposed model. Precision measures the model’s ability to present only correct entities, while Recall measures the model’s ability to identify all entities in the data set. F1 is the harmonic average of Precision and Recall, which can be calculated as follows:
Precision = TP
TP + FP (5)
Recall = TP
TP + FN (6)
F 1 = 2 ∗ Precision ∗ Recall Precision + Recall
(7)
Table 1 shows the details of the Bert-Crf-Rdrop model hyper-parameters. The total training cycle of this model was set to 58, and the batch size was set to 4. The initial learning rate is set to 1 x 105. Taking into account the average length of sentences in the data set, the sequence length is 256. We also use Dropout in our approach for better performance in the open domain.
Due to the limitation of time and computing resources, we only give the results of the Chinese model. Table 2 shows the prediction results of the test set based on BERT model, BERT-CRF model and BERT-CRF-Rdrop model.
It can be seen that the BERT-CRF-Rdrop model achieves better performance in the response part of the test set compared with BERT and BERT-CRF, and significantly improves precision, recall rate and
F1 score. A total of 22 teams participated in the China leg of the SemEval-2023 MultiCoNER Task, and each team submitted at least one entry. Our best submission achieved an F1 score of 0.5657, placing 15th out of 22 teams.

heading: 6 Conclusion
text: This study presents the BERT-CRF-Rdrop system submitted for the SemEval2023 Task 2: MultiCoNER Chinese track, including system design, implementation and evaluation. To address the challenges of complex, fuzzy, and emerging entity problems, we used BERT embedding to represent each character in the original sentence and trained CRF-Rdrop to predict named entity categories using the data set provided by the organizer. The experimental results show that our pre-training language model is effective in named entity recognition. In future work, we will further improve performance by developing more complex integration strategies and more diverse models.

heading: Future Work
text: In future work, we will further improve performance by developing more complex integration strategies and more diverse models