L3i++ at SemEval-2023 Task 2: Prompting for Multilingual Complex Named Entity Recognition
Year: 2023

Abstract:
This paper summarizes the participation of the L3i laboratory of the University of La Rochelle in the SemEval-2023 Task 2, Multilingual Complex Named Entity Recognition (MultiCoNER II). Similar to MultiCoNER I, the task seeks to develop methods to detect semantic ambiguous and complex entities in short and low-context settings. However, MultiCoNER II adds a fine-grained entity taxonomy with over 30 entity types and corrupted data on the test partitions. We approach these complications following prompt-based learning as (1) a ranking problem using a seq2seq framework, and (2) an extractive question-answering task. Our findings show that even if prompting techniques have a similar recall to fine-tuned hierarchical language model-based encoder methods, precision tends to be more affected.

heading: 1 Introduction
text: In SemEval-2022, Multilingual Complex Named Entity Recognition (MultiCoNER) I (Malmasi et al., 2022a) (Task 11) aimed at developing complex NER systems for 11 languages. The task focused on detecting semantically ambiguous and complex entities in short, lowercase, low-context monolingual, and multilingual settings. These challenges needed to be approached, while testing the domain and language adaption capability of the NER systems proposed by the participants. Thus, our participation consisted of a robust approach for tackling the following challenges for the language-specific, multilingual, and mixed sub-tasks (Boroş et al., 2022): (1) short texts: we built multilingual contexts with and without entities, in order to improve the focus of the NER system more on the context and less on the other surrounding entities; (2) lowercase texts: we prioritized the usage of uncased language models; and (3) code-mixed and
low-resource languages: we added a multilingual language model along with a language-specific language model. Our findings showed that, while adding contexts (from train partitions), with and without entities, was promising, the topics or domains overlap influenced the performance in both directions.
In SemEval-2023, MultiCoNER II (Fetahu et al., 2023b) (Task 2) featured complex NER in these languages (slight changes in comparison with the previous MultiCoNER): English, Spanish, Hindi, Bangla, Chinese, Swedish, Farsi, French, Italian, Portuguese, Ukrainian, and German. In addition to multilingualism, there were two new key challenges: (1) a fine-grained entity taxonomy with over 30 different classes, and (2) simulated errors added to the test set to make the task more realistic and difficult. Approaching these challenges would require an ability to easily adapt to the taxonomies of entities that are constantly evolving and to new scenarios with noise (for which there are few or no labelled data). Recently, a new paradigm based on prompt-based learning (prompt engineering) has achieved great success in information extraction tasks by reformulating classification tasks as close questions (Li et al., 2019; Liu et al., 2022; Boros et al., 2022).
Inspired by the advent of prompt engineering (Liu et al., 2023) or specifically templatebased prompting methods (Cui et al., 2021; Liu et al., 2022), we propose to treat the NER task as (1) a ranking problem using the sequence-tosequence (seq2seq) framework, and (2) an extractive question-answering (QA) task. In the scope of our research (and due to several constraints), we focused only on five languages.
In Section 2, we detail the data in the five languages we chose to participate in this year and the prompt engineering-based methods for approaching NER in Section 3. We then point out our findings and perform an error analysis in Section 4. We
807
present several similar works in Section 5. Finally, in Section 6 we present our conclusions, future work and discuss the limitations of the proposed methods.

heading: 2 Data
text: We utilized the MultiCoNER II corpus built by the organizers to challenge systems on the complex NER task (Fetahu et al., 2023a), where entities are not only standard entity classes such as locations, persons, organizations, etc., but also titles of creative works or medical-related terms. The corpus comprises documents in twelve languages and a multilingual dataset corresponding to the concatenation of all monolingual datasets. Entities are organized using a taxonomy of two levels, with six entries at the top level (coarse-grained) and 36 at the bottom level (fine-grained). However, in practice, only 33 are used.
Table 1 presents the dataset statistics for the five languages (Bangla, German, English, Hindi, and Chinese) and the multilingual dataset in which we participated. Test partitions are generally twice as large as train partitions, except for the English dataset, where the test partition is almost 15 times larger than the train one. For all datasets, train partitions are 19 times larger than the development ones. We detected the existence of repeated sentences at different rates among all train partitions. A detailed description and statistics of the MultiCoNER II corpus can be consulted in Fetahu et al. (2023a).

heading: 3 Methodology
text: In this section, we investigate the repercussion of semantically ambiguous and complex named entities on two prompt-based NER methods and their feasibility to be used in noisy and multilingual settings. To position the performance of the explored methods, we set a strong baseline, which has shown to be effective even on noisy datasets.

heading: 3.1 Stacked Transformers (StackedNER)
text: Our baseline model is the StackedNER architecture that corresponds to the BERT+2×T model (Boroş et al., 2022) from our previous participation in MultiCoNER I (Malmasi et al., 2022b). StackedNER comprises a hierarchical structure, with a pretrained and fine-tuned BERT-based encoder. The encoder consists of a stack of Transformer blocks and a conditional random field (CRF) prediction layer that is applied to decode the best tag path from all possible tag paths. Enhancing the model by leveraging transfer learning with additional Transformer layers has proven to be effective in handling noisy and varied input lengths, as well as detecting coarse-grained and fine-grained named entities (Boros et al., 2020a,b).
Hyperparameters We used the same hyperparameters as by Boroş et al. (2022). As for the pre-trained language model, we used bert-basemultilingual-uncased1.

heading: 3.2 Templates for NER (TemplateNER)
text: The first prompt-based NER method considers the task as a ranking problem using the sequence-tosequence (seq2seq) framework, where the original sentences are the source sequence and the templates filled by the candidate named entities span are the target sequence during the training. During inference, each candidate span is classified based on the template score.
We chose BART (Lewis et al., 2019), a denoising autoencoder for pretraining seq2seq models and fine-tuned it for English, while for other languages, we applied mBART (Liu et al., 2020), the multilingual version that was pre-trained on large-scale monolingual corpora in many languages using the same BART objective. The templates that we proposed are shown in Figure 1.
Furthermore, we tested the scenarios of using an English template and apply to another language
1https://huggingface.co/bert-base-multilingual-uncased
(e.g., German) as well as manually selecting different templates for a language (some variants include (1) is a/an (2) entity or (1) should be tagged as (2)). We also tested with different templates and transfer templates from one language to another one. The results are demonstrated in Table 2. The design of templates helps the models better fit our specific task. Further approaches to selecting the proper templates (e.g., gradient-based discrete searching, language model generating) will be left for future work.
Hyperparameters We experimented with BART2 (Lewis et al., 2019) for English and mBART3 (Tang et al., 2020) for the other languages, with a batch size of 32, 5 epochs, and a max sequence length of 70.

heading: 3.3 Question Answering for NER (QaNER)
text: The second approach treats NER as an extractive question-answering (QA) task (QaNER). Given a context C and a question Q, the goal is to extract
2https://huggingface.co/facebook/bart-large 3https://huggingface.co/facebook/mbart-large-50-many-
to-one-mmt
the answer A as a sub-string of C. Thus, a sample is composed of the triple (C, Q, A). Similar to Liu et al. (2022), we utilize a pre-trained BERT-based encoder to represent the concatenated Q and C. Then, a start and end token classifier layer maps the encoded text to a predicted span corresponding to the start and end of the answer A.
To obtain named entities from an extractive QA architecture, questions such as Where is the location?, Who is the person? or What is the product? are needed to be prompted. In these examples, the adverbs where, who, and what depend on the entity types location, person, and product. Liu et al. (2022) explored different ways of constructing these prompts and concluded that manually deciding which adverb must precede an entity type produced better results for coarse-grained taxonomies with no more than 12 classes. The case of the MultiCoNER II corpus with more than 30 entity types is different. Our preliminary experiments over the English dataset showed that automatically prompting the template [MASK] is the [ENTITY] ? ([ENTITY] is replaced by each one of the 36 fine-grained entity types) into the BERTbased encoder to obtain the corresponding 5W1H (i.e., who, what, when, where, why) question word produced the best results. We thus decided to follow this procedure to perform all our experiments.
To train our QA system, we first transformed each phrase from the MultiCoNER II corpus into multiple triples (C, Q, A), one for each entity type. It is possible for a phrase to have multiple answers for one entity type, in those cases, multiple triples for the same question are needed. For those entity types not present in the phrase, we assigned the classification [CLS] token for the start and end of the answer span. At the end of this process, we obtained at least 36 triples from one phrase of the original training set. Then, for each sample (C, Q, A) with Q and C concatenated by [CLS],
the training procedure consists of maximizing the likelihood that two tokens located at the right side of [CLS] correspond to the beginning and end of the answer A.
During entity prediction of a given phrase C, we first concatenated the corresponding Q of all entity types with C, then we prompted the system to obtain the tokens indicating the beginning and end of each entity type or the [CLS] token representing the non-existence of that named entity in the phrase. Multiple answer spans were possible for one entity type, therefore we checked for overlaps and retain the spans that obtained the highest prediction scores without overlapping. When two or more entity types were predicted for the same span, we kept the entity type with the highest score.
Hyperparameters We experimented with the multilingual XLM-RoBERTa4 (Conneau et al., 2019) pre-trained model for all languages except English in which we used BigBird5 (Zaheer et al., 2021). We set to 64 tokens the maximum sequence length with a stride of 16, learning rate to 2×10−5, weight decay equal to 0.01, and 4 epochs.
4https://huggingface.co/xlm-roberta-base 5https://huggingface.co/google/bigbird-roberta-base

heading: 4 Results
text: The performance of the systems for the NER task was evaluated using macro precision (P), recall (R), and F-score (F1). Tables 3 and 4 display the performance of all systems for both fine- and coarsegrained entities over the development and the test partitions, respectively.

heading: 4.1 General Observations
text: StackedNER ranks first for most of the results on the development partition for the coarse-grained (five out of six). Moreover, QaNER manages to outperform StackedNER in half of the languages for the fine-grained entities. TemplateNER does not perform well regardless of the language or the kind of entity. However, recall performances are interestingly dominated by QaNER in coarse- and fine-grained for most languages, except for Chinese and Multilingual. From these results, we can evidently conclude that StackedNER is a strong baseline, but QaNER is an interesting alternative that may be combined with a more precise NER to positively impact the final result. Similar results are observed on the test partition (Table 4), but with a clear dominance by StackedNER in all
configurations.

heading: 4.2 Error Analysis
text: Impact of Entity Type One new challenge proposed in MultiCoNER II is the introduction of a fine-grained entity taxonomy with 33 different classes. From Tables 3 and 4 we can conclude that this class diversity clearly impacts the systems’ performance, especially in terms of precision. To deepen the analysis, we display in Figure 2 the behaviour of QaNER for each class, and language in terms of precision, recall and F1.
We can observe that classes such as OtherLOC, Sympton, AnatomicalStructure, MedicalProcedure, OtherPROD, Drink, Food, Vehicle, Clothing, OtherPER, and ArtWork are problematic for all languages and present precision scores lower than 0.5. The case of Bangla is interesting, the system gives the impression of being precise for PrivateCorp (a precision of 0.750), however, it misses more than 83% of the entities corresponding to this class. On average, HumanSettlement, SportsGRP, and Station obtained the best scores among all languages in terms of F1, while PrivateCorp, Sympton, and Artwork were poorly classified.
Impact of Noise The introduction of noisy subsets for some of the languages (i.e., English and Chinese) consisted in the corruption of context or entity tokens in 30% of the sentences. Figure 3 shows the performance of the QaNER system for clean and noisy subsets in terms of F1. We observe that the noisy tokens have a negative impact on both languages, nevertheless, it is more evident in Chinese. A clear example is the ArtWork entity type, which decreases abruptly from 0.303 to 0.028. We interpret this behaviour as the result of two factors. First, 82% of the noisy tokens are present on entity tokens, compared to 76% for English. Second, QaNER obtained the lowest performance for Chinese with clean sentences, showing its limited capacity to handle it.
Impact of Domain Shift MultiCoNER is focused on challenging NER models with complex named entities, multilingualism, and cross-domain adaptation capabilities (Malmasi et al., 2022b; Fetahu et al., 2023b). To quantify the domain distribution among partitions, we obtained topic vectors by a joint representation of sentences and token semantic embeddings with Top2Vec (Angelov, 2020). We first projected all words and sentences
into a common multidimensional space with a multilingual Sentence-BERT (Reimers and Gurevych, 2020) pre-trained model (paraphrase-multilingualMiniLM-L12-v26).
Then, to reduce dimensionality, we applied UMAP to the sentence vectors (McInnes et al.,
6https://huggingface.co/sentencetransformers/paraphrase-multilingual-MiniLM-L12-v2
2018) followed by HDBSCAN (Campello et al., 2013) to find dense areas of sentences. Finally, to obtain the topic vectors, we calculated the centroid of each dense area. A topic is therefore characterized by the words and sentences present in the vicinity of its centroid. We kept the same hyperparameters as set by Boroş et al. (2022). From Tables 1 and 5 we can notice that the topic variation present on each partition depends on the number of sentences that constitute them.
In order to measure the presence of out-ofdomain sentences between data partitions, we computed the topic overlap between (train & dev) and (train & test) partitions. First, for each topic vector in a train partition, we calculated its cosine similarity against all topic vectors from development and test partitions. We then compared the topics by calculating the mean of the cosine similarities
between the sets of topics. The resulting topic similarities are shown in Figure 4. Multilingual presents the smallest similarities, which is an expected behaviour given the mixture of languages and the number of sentences. Bangla presents the highest topic similarity while having the smallest number of topics. This shows uniformity in the distribution of its topics which has shown to be beneficial for certain entity types such as PrivateCorp, ORG and Station.

heading: 5 Related Work
text: Similar to our work, Chen et al. (2022) approached NER as a generation problem and introduced LightNER, a tuning paradigm for NER that adopts a pluggable prompting method to improve NER performance in low-resource settings by freezing most of the generation model parameters. Their experimental results showed that LightNER obtained comparable performance in the standard supervised setting and outperformed strong baselines in low-
resource settings. Template-based NER was explored by Cui et al. (2021), where NER was approached as a language model ranking problem under a sequenceto-sequence framework (generative pre-trained language model). The source sequences corresponded to the original sentences, while the target sequences were statement templates filled by the candidate named entity spans. During entity prediction, the model classified each span based on the corresponding template scores. Their experiments demonstrated that the proposed method performed significantly better than fine-tuning BERT.
Concerning QA, Liu et al. (2022) explored a prompted-based learning NER method based on extractive QA. In this approach, each entity type was formulated into a question and prompted into the NER system to obtain the offset within the input context corresponding to the tokens of the entity type. The method showed to be faster and more robust in low-resource conditions compared to other prompting methods.

heading: 6 Conclusions
text: In this paper, we investigated the performance of two prompt-based NER methods over five monolingual and multilingual datasets provided by the SemEval-2023 Task 2, Multilingual Complex Named Entity Recognition (MultiCoNER II). Our findings show that semantically ambiguous and complex entities in short and low-context settings are challenging for prompting techniques, affecting mostly their precision. These techniques were also influenced by the entity type taxonomy and context size, which increased the training and inference time. In future work, we will explore other methods to select proper templates and include knowledge bases to add informative context that helps discern the correct entity type.
Limitations
The TemplateNER approach has several limitations due to two aspects. First, it is time-consuming to generate the template for all the possible spans at the sentence level. Here, we only consider the named entities that constitute up to four words (1-4 grams) to reduce the time cost of the models. As the number of entities in a sentence is much fewer than non-entities, there is a significant imbalance between the number of templates enumerated for entities and the non-entities, which might affect the
performance. Second, if the length of the sentence increases, the decoding time would rise catastrophically. Further improvements are left for future development.
Regarding QaNER, the number of samples is multiplied by the number of entity types, which could be expensive, especially for cases when a fine-grained entity prediction is needed. It is also important to take into consideration the word or words used to replace [ENTITY] in the template [MASK] is the [ENTITY]?. Entity types such as ArtWork present possible substitutions (i.e., [MASK] is the artwork?, [MASK] is the work of art?) which have an impact on the encoding done by the pre-trained language model.
Ethics Statement
We noticed that MultiCoNER II has taken a political stance. Due to this reason and also, the size of some language-specific datasets, we did not participate in some of the languages.

heading: Acknowledgements
text: This work has been supported by the ANNA (2019-1R40226) and TERMITRAD (2020-2019- 8510010) projects funded by the NouvelleAquitaine Region, France. We would also like to thank our colleagues Wenjun Sun, Nicolas Sidère, Mickaël Coustaty, Beatriz Martı́nez Tornés, and Souhail Bakkali.

heading: Future Work
text: In future work, we will explore other methods to select proper templates and include knowledge bases to add informative context that helps discern the correct entity type. Limitations
The TemplateNER approach has several limitations due to two aspects. First, it is time-consuming to generate the template for all the possible spans at the sentence level. Here, we only consider the named entities that constitute up to four words (1-4 grams) to reduce the time cost of the models. As the number of entities in a sentence is much fewer than non-entities, there is a significant imbalance between the number of templates enumerated for entities and the non-entities, which might affect the
performance. Second, if the length of the sentence increases, the decoding time would rise catastrophically. Further improvements are left for future development. Regarding QaNER, the number of samples is multiplied by the number of entity types, which could be expensive, especially for cases when a fine-grained entity prediction is needed. It is also important to take into consideration the word or words used to replace [ENTITY] in the template [MASK] is the [ENTITY]?. Entity types such as ArtWork present possible substitutions (i. e. , [MASK] is the artwork?, [MASK] is the work of art?) which have an impact on the encoding done by the pre-trained language model