PAI at SemEval-2023 Task 2: A Universal System for Named Entity Recognition with External Entity Information
Year: 2023

Abstract:
The MultiCoNER II task aims to detect complex, ambiguous, and fine-grained named entities in low-context situations and noisy scenarios like the presence of spelling mistakes and typos for multiple languages. The task poses significant challenges due to the scarcity of contextual information, the high granularity of the entities(up to 33 classes), and the interference of noisy data. To address these issues, our team PAI proposes a universal Named Entity Recognition (NER) system that integrates external entity information to improve performance. Specifically, our system retrieves entities with properties from the knowledge base (i.e. Wikipedia) for a given text, then concatenates entity information with the input sentence and feeds it into Transformer-based models. Finally, our system wins 2 first places, 4 second places, and 1 third place out of 13 tracks. The code is publicly available at https://github. com/diqiuzhuanzhuan/semeval-2023.

heading: 1 Introduction
text: The objective of the MultiCoNER II shared task (Fetahu et al., 2023b) is to develop a robust named entity recognition (NER) system for multiple languages (including English, Spanish, Hindi, Bangla, Chinese, Swedish, Farsi, French, Italian, Portuguese, Ukranian, German and aforementioned mixed languages) that can perform well even when the input data has spelling errors, lacks contextual information, or contains out of knowledge base entities. This task entails the recognition of a total of 33 fine-grained entity types within six coarsegrained entity categories. The entity types within each category are highly prone to confusion and ambiguity, and the testing data includes noise such as spelling errors that must be considered. Similar to the previous competition (Malmasi et al., 2022b), the data in this competition contains a lot of short
*These authors contributed equally to this work.
and low-context sentences, which means the accurate prediction of entity types relies heavily on external knowledge.
Therefore, the key objective of our current work is to achieve effective integration of the model and external knowledge. In this paper, extending our previous work (Ma et al., 2022), which has demonstrated the efficacy of the Dictionary-fused model, we propose a NER system based on Entity Property Knowledge Base. We constructed our knowledge base using all entities and their associated properties (including entity name, alias, sitelink title, description, instanceof, subclassof and occupation) from WikiData. In the retrieval module, we use string matching to retrieve entities with properties for a given sentence. Given the sequence length limitations of our model, we keep longer entities in sentences. Subsequently, we put the entity and property values into the external context in a specific format and feed the external context and original sentence into the NER model. Figure 1 illustrates our motivation, demonstrating how the inclusion of property values can effectively improve performance in practice.
We further propose entity-aware attention mechanism, which can better model the semantic relationships between entities in sentences and external context than conventional attention. In the test phase, we use the voting strategy to form the final predictions.
We make the following observations according to our experiments:
1. The system based on entity property knowledge base has significantly improved the effect. The knowledge base has a high entity coverage, and even more than 95 percent in some low-resource languages such as Bangla and Hindi.
2. Different properties have different effects on different entity types. Specifically, occupation
744
property can improve the recognition of finegrained person entities.
3. We have observed that the dictionary can greatly improve the performance on clean data, but may have a negative impact on noisy data. Knowledge retrieved by fuzzy matching such as ElasticSearch can help identify noisy entity.
4. On some tracks, our entity aware attention can better capture the semantic relationships between entities in sentence and external context.

heading: 2 Related Work
text: Complex named entities (NE), like the titles of creative works, are not simple nouns and pose challenges for NER systems (Malmasi et al., 2022a; Fetahu et al., 2023a). To mitigate these challenges, NER models utilizing external knowledge have achieved remarkable results. Researchers have integrated gazetteers into models (Bender et al., 2003; Malmasi and Dras, 2016) in earlier studies. Recently, GEMNET (Meng et al., 2021) proposes a novel approach for gazetteer information integration, where a flexible Contextual Gazetteer Representation (CGR) encoder can be fused with any word-level model and a Mixture-of-Experts (MOE) gating network can overcome the feature overuse issue by learning to conditionally combine the context and gazetteer features, instead of assigning them fixed weights. GAIN (Chen et al., 2022)
adapts the representations of gazetteer networks to integrate external entity type information into models. (Ma et al., 2022) uses string matching to retrieve entities with types from the gazetteer and concatenates the entity information with the input text and feds it to models. However, the prerequisite for these approaches is to construct an entity dictionary with accurate types, which is extremely difficult to fulfill. Knowledge-fused methods mentioned in (Wang et al., 2022; Çarık et al., 2022) build an information retrieval (IR) system based on Wikipedia to provide related context information to models. But these methods require retrieving high-quality context, which is difficult to guarantee in the MultiCoNER II shared task (Fetahu et al., 2023b).
Actually, Transformer-based models (Devlin et al., 2019; Vaswani et al., 2017; Yamada et al., 2020) have the ability to automatically learn the mapping between the entity properties and entity types on the basis of input sentence. Therefore, We propose a NER system that fuses entity dictionaries without accurate entity types to overcome these drawbacks of previous work.

heading: 3 Our System
text: In this section, we introduce how our system works. Given a sentence x = {x0, x1, ..., xn−1}, x consists of n tokens, and this sentence is input to the knowledge retrieval module. The retrieval module outputs the retrieved entities and property contexts. The system then concatenates the input sentence and outputs from the retrieval module and fed them to a Transformer-based model. To enhance the relevance of retrieved entities to the given information, we devised entity-aware attention mechanism. The overall architecture is shown in Figure 2.

heading: 3.1 Knowledge Retrieval Module
text: Knowledge Base Construction The entity property KB is constructed based on WikiData. Wikidata contains more than 90 million different entities. Two entity examples with related fields are shown in Table 1. Qid represents an entity, and Labelqid, Aliasqid, Sitelink_titleqid, Subclassofqid, Instanceofqid and Occupationqid represents the values of different fields of the entity. Entity_namesqid represents the entity names, consists of Labelqid, Sitelink_titleqid and Aliasqid. Property_qidqid represents the property values in qid format,
consists of Subclassofqid, Instanceofqid and Occupationqid. Property_namesqid represents the property names in text format, which is the label field values of Property_qidqid. Contextqid represent the entity context. In Entity_contextqid, Property_namesqid is separated by a special token "|". Two entity examples with names and context are shown in Table 2. The knowledge base contains all entity names and contexts.
Entity Retrieval Given an input sentence, the retrieval module retrieves entity names by string matching. The external context consists of matched entity names and corresponding entity contexts. We prioritize preserving the longer entities when
encountering overlapping matches or the need to truncate information. Finally, retrieval module returns multiple entity context pairs, pairs = [(entity0, context0), (entity1, context1), ...]. The length of the pairs is m.

heading: 3.2 Entity Aware Attention
text: Our system concatenates input sentence x and entity contexts together, and entity contexts are separated by a special token "$".
In order to better use the external knowledge in Transformer-based model, entity-aware attention is proposed to link entities in sentence with corresponding contexts. Entity-aware attention
matrix M is a binary matrix with values 0 or 1, which makes context only attended by the corresponding entity. Under both conditions Mi,j = 1, other times Mi,j = 0. In the first condition, i, j < n+2. n+2 takes into account [CLS] token at the front of the sentence and [SEP] token at the end of the sentence. In the second condition, i in positionentityk , and j in positioncontextk , 0 <= k < m. positionentityk and positioncontextk are position subscript sets of entityk and contextk
respectively. Entity-aware attention matrix M is input into Transformer-based NER model as attention mask. An entity-aware attention matrix example is shown in Figure 2.

heading: 4 Exprimental Setup

heading: 4.1 Data and Evaluation Metrics
text: MultiCoNER II contains 6 coarse-grained labels, which can be further subdivided into 33 finegrained labels. The specific labels are shown in table 8. Fine-grained named entity recognition (NER) is a particularly challenging task, especially in low-context and ambiguous situations. For instance, identifying names of individuals who are scientists or musicians can be super difficult without sufficient context.
The task has 12 monolingual tracks, including multiple low-resource language tracks that are difficult to obtain external knowledge, such as Hindi and Bangla. In this paper, all results use entity level micro F1 as evaluation metric except table 5. Table 5 use macro F1 which is official evaluation metric.

heading: 4.2 Training detail
text: We use Chinese BERT with Whole Word Masking (Cui et al., 2019) as our pre-trained model in Chinese and utilize BERT multilingual base model (uncased)(Wolf et al., 2020) in other tracks. An 8-fold cross-validation training strategy is applied
in the evaluation except for MULTI task. In the evaluation phase, all the best models vote on the prediction results, and the voting weights are determined by the F1 score of each model on the validation set.

heading: 5 Results and Analysis

heading: 5.1 Main Results
text: To show the effectiveness of our system, we evaluate the results of a baseline system without knowledge. We reproduce the dictionary construction process of (Chen et al., 2022) on MultiCoNER II, and obtain a type dictionary through manual relational mapping. The results of the baseline, Manual Type dictionaries, and our system are shown in Table 4. The automatic property dictionary exceeds the baseline by 12.2 F1 on average, demonstrating the effectiveness of our system. Without manual relational mapping, our proposed property dictionary outperforms the manually constructed type
dictionary by 7.8 F1 on average, demonstrating the capability of the model to learn the correlation between property values and entity types automatically.

heading: 5.2 Coverage Rate Trial
text: Previous studies(Rijhwani et al., 2020; Meng et al., 2021) have found that systems with higher entity coverage have higher performance. Table 3 shows the entity coverage of Manual Type Dictionary and Property Dictionary. The coverage rate of the property dictionary is 44 percent higher than that of the artificial dictionary and even reaches 99 percent in some languages, such as Chinese and Hindi. The track with higher coverage rate has bigger improvement in Table 4.

heading: 5.3 Effect of Property
text: In order to analyze the impact of different properties on the results, we conduct ablation experiments of properties. Table 6 shows the results of using different properties. We observe that the Occupation property has a greater impact on the performance. Without Occupation property, the average results drop by 4.5 F1. Further, we analyze the relationship between properties and taxonomies. Table 8 shows the taxonomy F1 score when using different property combinations. The subclassof property is strongly related to product class. The instanceof property is strongly correlated with Creative Works, Group, Location, and Medical classes. Occupation is a property about people. Experiments show that occupation property plays a crucial role in identifying Person class.

heading: 5.4 Effect of Entity Aware Attention
text: We propose entity-aware attention, which can represent the relationship between entities and context in
a sentence. Table 7 shows the effect of entity-aware attention on several languages. The entity-aware attention improves by 1 F1 and 0.3 F1 over the baseline on Bangla and Italian, demonstrating its effectiveness.

heading: 5.5 Clean Data and Noisy Data
text: Table 5 shows the results of our system on clean data, noisy data, and overall data in evaluation. The results demonstrate that our system achieves higher performance on clean data than on noisy data, which poses a challenge due to the noisy entities that can not be retrieved through string matching. The Chinese track exhibits the largest discrepancy between clean and noisy data since the Chinese training and validation sets only contain clean data, while the test set includes both clean and noisy data.

heading: 6 Conclusion
text: In this paper, we describe our NER system based on entity properties, which wins two tracks in MultiCoNER II shared task. We construct a KB based on entity properties, which is used to retrieve the relevant entity names and contexts for a given sentence. Our property dictionary is built without the need for manual relational mapping and achieves high coverage on the test set. We have found that different entity types require different properties. We propose the entity-aware attention mechanism to better learn the relationship between entities and contexts. In the future, we plan to adopt fuzzy matching to improve the performance on noisy data and explore our system for other low-resource tasks.

heading: Future Work
text: In the future, we plan to adopt fuzzy matching to improve the performance on noisy data and explore our system for other low-resource tasks