USTC-NELSLIP at SemEval-2023 Task 2: Statistical Construction and Dual Adaptation of Gazetteer for Multilingual Complex NER
Year: 2023

Abstract:
This paper describes the system developed by the USTC-NELSLIP team for SemEval2023 Task 2 Multilingual Complex Named Entity Recognition (MultiCoNER II). A method named Statistical Construction and Dual Adaptation of Gazetteer (SCDAG) is proposed for Multilingual Complex NER. The method first utilizes a statistics-based approach to construct a gazetteer. Secondly, the representations of gazetteer networks and language models are adapted by minimizing the KL divergence between them at both the sentence-level and entity-level. Finally, these two networks are then integrated for supervised named entity recognition (NER) training. The proposed method is applied to XLM-R with a gazetteer built from Wikidata, and shows great generalization ability across different tracks. Experimental results and detailed analysis verify the effectiveness of the proposed method. The official results show that our system ranked 1st on one track (Hindi) in this task.

heading: 1 Introduction
text: Named Entity Recognition (NER) is a fundamental and important natural language processing (NLP) task, which aims at finding entities and recognizing their type in a text sequence. Recently, deep neural networks have achieved great performance on simple NER with abundant labeled data (Ye and Ling, 2018; Jia et al., 2020; Chen et al., 2022). In practical and open-domain settings, it is difficult for machines to process complex and fine-grained named entities (Ashwini and Choi, 2014; Fetahu et al., 2023b). For example, “The Old Man and the Sea” is the title of a movie as well as a book, which has different categories in different contexts and cannot be recognized easily by present NER systems. This issue may become even more serious in multilingual settings (Fetahu et al., 2021). However, it has not received sufficient attention from the research community. To alleviate the
issue, SemEval-2023 Task 2 (Fetahu et al., 2023b) formulates this task which challenges participants to develop NER systems for 12 languages (English, Spanish, Swedish, Ukrainian, Portuguese, French, Farsi, German, Chinese, Hindi, Bangla and Italian), focusing on recognizing semantically complex and fine-grained entities in short and low-context settings. Each language constitutes a single track, while Multilingual is added as Track 13. The datasets (Fetahu et al., 2023a) mainly contain sentences from three domains: Wikipedia, web questions and search queries. Besides, simulated errors are added to the test set to make the task more realistic and difficult.
Recent studies have found that integrating external knowledge or gazetteers into neural architectures is effective in solving this problem (Liu et al., 2019; Rijhwani et al., 2020; Meng et al., 2021). For example, the two representations respectively from a language model and a gazetteer network are integrated as one representation, which is further fed into a classifier such as a conditional random field (CRF) (Lafferty et al., 2001). However, for the fine-grained entities, due to the closer semantic distance between these entity categories, the coverage rate of the constructed entity gazetteer is difficult to improve. Besides, the interaction between the two representations in previous work (Chen et al., 2022) only focus on sentence-level, which ignore the entity-level representation gap between them and lead to information loss.
In this paper, we propose a method named Statistical Construction and Dual Adaptation of Gazetteer (SCDAG) for Multilingual Complex NER based on GAIN (Chen et al., 2022). Firstly, based on Wikipedia of the 12 languages, we build a multilingual gazetteer to search for the entities in input sentence. Different from GAIN, we use a statistics-based approach to maximize the coverage of the gazetteer. Afterwards, the SCDAG adopts a two-stage training strategy to dually
651
adapt the gazetteer network to the language model. During the first training stage, the parameters of a language model are fixed. Then a sentence and its annotation are fed into the two networks separately. The representations of gazetteer networks and language models are adapted by minimizing the KL divergence between them at the sentence-level and entity-level. This process helps the gazetteer network understand the meaning of NER tags and strengthen the model adaptation ability to NER. A gazetteer is applied to sentences to generate pseudo tags which are fed into the two pre-trained networks separately in the second stage. Finally, the two output representations are integrated for classifying.
The proposed method achieves great improvements on the validation set (Fetahu et al., 2023a) of SemEval-2023 Task 2 compared to baseline models with gazetteers. Ensemble models are used for all thirteen tracks in the final test phase, and our system officially ranked 1st on one track (Hindi). The outstanding performance demonstrates the effectiveness of our method. To facilitate the reproduction of our results, the code is available at https://github.com/mjy1111/SCDAG.

heading: 2 Related Work
text: NER has a lot of applications in various domains and languages. Recently, with the introduction of contextual pre-trained models, such as BERT, ROBERTA (Delobelle et al., 2020) and XLMR (Conneau et al., 2020), the performance of NER systems has been significantly improved. These models are trained on large-scale unlabeled data such as Wikipedia, which can significantly improve the contextual representations abilities.
SemEval-2023 Task 2 is a continuation of the multilingual NER task started in 2022 (Malmasi et al., 2022b). There are many challenges that can make NER extremely difficult. In Meng et al. (2021), they explain that named entity recognition is especially difficult in situations with low-context or in scenarios where the named entities are exceptionally complex and ambiguous. Another work has extended this to multilingual and codemixed settings (Fetahu et al., 2021). These are the key challenges of the 2022 datasets (Malmasi et al., 2022a) and we have participated in this competition (Chen et al., 2022). Besides, NER requires abundant well-annotated data, which is too expensive in low-resource languages (Ma et al.,
2022). Lots of methods are proposed to improve NER performance and a general discovery is that the use of external knowledge bases is very effective. Wang et al. (2021) retrieve related contexts from a search engine as external contexts of the inputs to take advantage of long-range dependencies for entity disambiguation and successfully achieve stateof-the-art performance across multiple datasets. Meng et al. (2021) recognize the importance of gazetteer resources, even in the case of state-ofthe-art systems making use of pre-trained models. They propose a Contextual Gazetteer Representation encoder, combined with a novel Mixtureof-Expert (MoE) gating network to conditionally utilize gazetteer information. Fetahu et al. (2021) employ multilingual gazetteers embedded with transformer models in an MoE approach to improve the recognition of entities in code-mixed web queries, where entities are in a different language from the rest of the query.

heading: 3 Data
text: The MultiCoNER dataset (Fetahu et al., 2023a) is provided in a column-based format and divided into the training, development and testing sets. The text is lowercase with the named entity annotation in BIO format (Sang and Meulder, 2003). The first token (or the single token) of an entity contains the “B-” prefix. Other entity tokens (in the case of multi-token entities) start with an “I-” prefix, while non-entity tokens are denoted with “O”. It consists of 6 coarse-grained entity types and 33 fine-grained entity types, and the coarse to fine level mapping of the tags is as follows: Location (Facility, OtherLOC, HumanSettlement, Station), Creative Work (VisualWork, MusicalWork, WrittenWork, ArtWork, Software), Group (MusicalGRP, PublicCORP, PrivateCORP, AerospaceManufacturer, SportsGRP, CarManufacturer, ORG), Person (Scientist, Artist, Athlete, Politician, Cleric, SportsManager, OtherPER), Product (Clothing, Vehicle, Food, Drink, OtherPROD), Medical (Medication/Vaccine, MedicalProcedure, AnatomicalStructure, Symptom, Disease).
Two data augmentation methods are used following (Chen et al., 2022). For the basic training set provided officially, an entity replacement strategy is adopted using our own gazetteer to construct a data-augmented set. This part of data is called “data-wiki”, which mainly consists of rich-context
where to buy apple iphone 13 O O O B-PROD I-PROD I-PROD
O O O B-CORP B-PROD I-PROD O O O B-PROD I-PROD I-PROD
(True tags from the training data) Gazetteer
PROD: apple iphone 13, iphone 13, apple CORP: apple
string match
(Searched noisy tags from the gazetteer)
Dense Layer
BiLSTM
XLM-Roberta
Gazetteer Network MLP MLP KL Divergence Integration Classifier Wikidata Construct Gazetteer-Adapted Integration Network （GAIN）
sentences. In order to improve the performance of our models on low-context instances, a set of annotated sentences are generated from the MSMARCO QnA corpus (V2.1) (Nguyen et al., 2016) and the ORCAS dataset (Craswell et al., 2020), which are mentioned in Meng et al. (2021). Our trained models and existing NER systems (e.g., spacy) are applied to identify entities in these corpora, and only templates identically recognized by all models are reserved. Then 3,379 English templates for MS-MARCO and 11,754 English templates for ORCAS are obtained. Next, we slot the templates by our own gazetteer and translate them to the other 11 languages. This part of data is called “data-query”. Finally, the constructed data is used together with the official data for training.

heading: 4 Methodology
text: This study focuses on making better use of external entity knowledge. To describe our system clearly, in this section, we first introduce three basic mainstream NER systems used. Then we show the process of constructing a gazetteer from Wikipedia using a statistics-based approach to maximize the coverage rate, and how the gazetteer representation is generated and utilized. Finally, we illustrate the dual adaptation between gazetteer network and language model. The overall structure of the proposed system is shown in Figure 1.

heading: 4.1 Basic NER Systems
text: We mainly use the XLM-RoBERTa large (Conneau et al., 2020) as the pre-trained language model, which is a widely used encoder. Generated by feeding a sentence into the encoder, the representation is then input to different classifiers. Three mainstream NER backend classifiers are adopted: Softmax (Devlin et al., 2019) and CRF (Huang
et al., 2015) are classic sequential labeling methods that predict the tag of each token, and Span (Yu et al., 2020) is a segment-based method that predicts the start and the end of an entity separately.

heading: 4.2 The Gazetteer
text: It’s difficult to process complex and fine-grained entities only relying on the language model itself (Ashwini and Choi, 2014). To integrate external entity knowledge, we first need to build a large gazetteer matching the taxonomy, then we have to consider how to fuse the gazetteer information with the semantic information from the language model.

heading: 4.2.1 Statistical Construction
text: Our gazetteer is built based on Wikidata. Wikidata is a free and open knowledge base. Every entity of Wikidata has a page consisting of a label, several aliases, descriptions, and one or more entity types. We use entity type annotated by Wikidata to construct the gazetteer. For example, “Da Vinci” can be annotated as a researcher or a well-known artist in Wikidata. Thus, according to the entity definition of this competition, the word “Da Vinci” is given both scientist and artist labels.
In the previous work (Chen et al., 2022), to construct a gazetteer fit to the data of this task, firstly every entity of the training set is searched in Wikidata. Then all the entity types returned are mapped to the NER taxonomy with 6 labels manually. Next, all Wikidata entities stored in these entity types can be added to the 6 labels gazetteer separately. However, since labels are fine-grained and some labels are semantically similar in this work, the cost of manual matching is too high and just adding the returned entity types to a certain label gazetteer will result in low coverage of the gazetteer. Therefore, we counted the coverage of the Wikidata entities contained in each returned entity type on each label of the training set, and added this entity type to the top two labels gazetteer with the highest coverage.
In this way, a multilingual gazetteer is constructed that contains entities from 70K to 3M for each language. The gazetteer approximately has a coverage rate of 35 percent on entities in the training and validation set. To validate the effectiveness of this method, we also use a manual one-to-one matching same as Chen et al. (2022). Basic information about our gazetteer is shown in Table 1. The coverage rate is calculated as the number of entities both appearing in the official data and our gazetteer divided by the total number of entities in the official data. It shows that with the statistic-based approach, the coverage rate of the gazetteer gains a significant improvement.

heading: 4.2.2 Application
text: A search tree is constructed for string matching firstly to apply the gazetteer to a sentence. Once a sentence is fed into the search tree, a maximum length matching algorithm will be conducted, and a 67-dimension one-hot vector for each token will be generated. Take the sentence “where to buy apple iphone 14” for example. By string matching with the gazetteer, “apple iphone 14”, “iphone
14” and “apple” are found in the OtherPROD gazetteer, while “apple” is also found in the Food gazetteer. Then a 67-dimension one-hot vector will be generated for every word as shown in Table 2.
Denote one sentence as w = (w1, w2, ...wM ) where M is the length of the sentence and wi is the ith word. By feeding w into the encoder such as the XLM-RoBERTa large, a semantic representation e ∈ RN×D is obtained, where N is the length of subword tokens and D is the hidden size. At the same time, the one-hot vector generated from the search tree is fed into a gazetteer network consisting of a dense layer and a BiLSTM. To match the hidden size of the language model, the output embedding g has the same size as e. Noting that the value of each word in gazetteer is assigned to the first subword, and the other subwords are 0.
Then we use two ways to integrate e and g. One way is to concatenate them on each token, another way is to get the weighted summation of them by setting a trainable parameter λ ∈ RN×D. The final representation is fed into the backend classifier for supervised NER training.

heading: 4.3 Dual Adaptation
text: Chen et al. (2022) have found that only conducting the normal training process above is not enough. Since the encoder XLM-R large and the gazetteer network BiLSTM are almost isolating each other during the whole training, almost no semantic information can be gained explicitly by the classic gazetteer network.
In this paper, the dual adaptation method is proposed with a two-stage training strategy to interact the representations of gazetteer network and language model at both sentence-level and entity-level. In the first stage, an one-hot vector is constructed just based on the true tags in training set for each sentence. A gazetteer representation gr ∈ RN×D is obtained after passing the vector
through the gazetteer network. Then the parameters of the language model are fixed, and the sentence w is fed into it to get a semantic representation s. {gr, s} are projected to {gtr, st} ∈ RN×67 by two separate linear layers, where the semantic meaning is transferred to the tags meaning as a kind of logits distribution. Meantime, all entity logits are connected and denoted as {ger, se} ∈ RE×67. Then the sentence-level and entity-level adaptation are respectively implemented by the designed losses:
L1 = KL(sg(gtr)||st) + KL(sg(st)||gtr), (1) L2 = KL(sg(ger)||se) + KL(sg(se)||ger), (2)
where KL(·) is the KL divergence calculation and sg(·) operation is used to stop back-propagating gradients, which is also employed in Jiang et al. (2020); Liu et al. (2020). The loss L1 encourages the distributions gtr and s
t to approximate each other to enhance the two networks as a whole at the sentence-level. The loss L2 utilizes entities to strengthen the model adaptation ability to NER. In the second stage, all the parameters are trained with a gazetteer. As illustrated in Section 4.2.2, a gazetteer representation g is generated from the search tree and the gazetteer network BiLSTM. Next, an ordinary fusion method is applied to g and s to get an integration representation, which is then fed into the backend classifier to compute a conventional loss with true tags T. This supervised training goal is implemented by the loss L3:
L3 = Classifier(f(g, s),T), (3)
where f(·) denotes ordinary integration methods like concatenation or weighted summation. Classifier(·) represents one of the three mainstream backend classifiers mentioned in Section 4.1. It is worth noting that in Eq. (2), we use the true entities in training set for the whole training. During the second-stage training, a multitask learning goal is conducted shown as:
L4 = α(L1 + L2) + L3, (4)
where α is a hyperparameter control the importance of gazetteer and is manually set for different fusion or backend methods.

heading: 5 Experiments

heading: 5.1 Implementation Details
text: In this paper, the XLM-R large model was chosen as the encoder for all tracks, which could be found on the HuggingFace Page (Wolf et al., 2019). As for hyperparameter, the hidden size was 1024, batch size was 32 and dropout rate was set to 0.1. The AdamW (Loshchilov and Hutter, 2019) was used as optimizer. We adopted a learning rate 2e-5 for language models, 2e-4 for gazetteer networks and 2e-5 for classifiers. The training epoch for firststage training was 5 and for second-stage training was 20. The α for the second stage training was set to 5 for Softmax and Span, 100 for CRF. All code was implemented in the PyTorch framework1.

heading: 5.2 Training Strategy
text: In this paper, a 5-fold cross-validation training strategy was also applied in the evaluation and a lot of models had been trained with the SCDAG method using different classifiers. Firstly, the prepared data “data-wiki” and “data-query” were split into five pieces, each one was used as the validation set, while the other four pieces were used as the training set. After obtaining the five best models by this strategy, the logits of them (for Softmax and Span models) were averaged to integrate them as an aggregated model. CRF models had been just voted averagely at the word level.
Finally, the predictions of our best models in different methods were token-voted by setting a weight for each track. The weight was manually set referring to all scores on the validation set.

heading: 5.3 Official Results
text: Our team participated in all 13 tracks and the overall fine-grained F1 and per-class performance were reported in Table 3. We also provided the coarse-grained metrics F1. We ranked 1st on the HI track. As shown in the table, the proposed SCDAG method significantly improved the performance of recognizing the fine-grained entities.

heading: 5.4 Analysis

heading: 5.4.1 Effectiveness of SCDAG
text: To explore the effectiveness of each module in the proposed SCDAG method, a large number of trials were conducted on the official data mentioned
1https://pytorch.org/
in Section 3. All scores under the concatenation integration setting on the validation set were listed in Table 4. Compared “Integration” with “Base”, significant improvements were gained by the gazetteer on all tracks. Compared “Integration” with SCDAG, we could find that the dual adaptation and the two-stage training strategy was effective for NER. Besides, the effect of the CRF method was stronger than that of Softmax, which may be because fine-grained classification was more difficult for linear classifiers.

heading: 5.4.2 Ablation Study on Constructed Data
text: To validate the contribution of the constructed data, the following variants were conducted to perform the ablation study on the test set: (1) SCDAG w/o. data-wiki, which removed the entity replacement strategy. (2) SCDAG w/o. data-query, which removed the templates of MS-MARCO and ORCAS corpus mentioned in Section 3.
The results of the ablation experiments were shown in Table 5. Some in-depth analysis could be explored: (1) Compared SCDAG with SCDAG w/o. data-wiki, the removal of the entity replacement strategy caused a significant performance drop, especially for EN, ZH, IT, ES, FR, PT and SV. This was because a noisy subset was held for each language where the sentences were corrupted with noise either on context tokens or entity tokens in the test set (Fetahu et al., 2023a). The “data-wiki” enhanced the robustness of the model. (2) Compared SCDAG with SCDAG w/o. dataquery, we could see that the removal of the templates caused a significant performance drop,
which further demonstrated the importance of introducing short sentence training data for lowcontext settings.

heading: 5.4.3 Average-Logits Experiments
text: This section explained why we chose to average logits of softmax-based models (for Softmax and Span models) for integrating them as an aggregated model, rather than an average token-vote. Also, a 5-fold cross-validation training was conducted with the official training data on the basic Softmax method. Without loss of generality, HI, EN, ZH and FA were chosen to represent different language families. The results of the official validation set were shown in Table 6. It was empirically demonstrated that average-logits for the softmaxbased model ensemble was better than averagetoken-vote in most situations.

heading: 6 Conclusion
text: This paper presents the implementation of the USTC-NELSLIP system submitted to the SemEval2023 Task 2 MultiCoNER II. Different from
MultiCoNER I, it has a fine-grained taxonomy which greatly increased the difficulty of the task. The SCDAG method is proposed to statistically construct gazetteer and dually adapt the gazetteer network to the language model, achieving great improvements on the fine-grained NER task. Some construction methods for gazetteers and augment data are also provided. In future works, we will improve the gazetteer quality and apply this method to more tasks.

heading: Acknowledgements
text: We thank anonymous reviewers for their valuable comments.
Limitations
Although the proposed method has shown great performance for MultiCoNER, this method still can be further improved. For instance, some textual enhancement could be adopted from Wikipedia so that the model could get stronger semantic information. Besides, some entity types in the gazetteer has a low coverage because statistical construction is not precise enough. In addition, since the gazetteer has a large number of irrelevant entities, denoising the gazetteer is worth studying.

heading: Future Work
text: In future works, we will improve the gazetteer quality and apply this method to more tasks