FII_Better at SemEval-2023 Task 2: MultiCoNER II Multilingual Complex Named Entity Recognition
Year: 2023

Abstract:
The “MultiCoNER II Multilingual Complex Named Entity Recognition” task1 at SemEval 2023 competition focuses on identifying complex named entities (NEs), such as the titles on creative works (songs, books, movies), in several languages. In the context of SemEval, our team, FII_Better, presents an exploration of a base transformer model’s capabilities regarding the task, focused more specifically on five languages (English, Spanish, Swedish, German, Italian). We take DistilBERT and BERT as two examples of basic transformer models, using DistilBERT as a baseline and BERT as the platform to create an improved model. In this process, we learned a lot about working with transformers and, on top of that, we managed to get fair results on the chosen languages.

heading: 1 Introduction
text: Named entity recognition (NER) is a subtask of natural language processing (NLP), being still a crucial learning problem (Lample et al. (2016), Zhang et al. (2022)). NER is used especially for relevant information extraction from text data (Shao et al. (2016), Gifu and Vasilache (2014)). It is impressive that state-of-the-art NER systems rely heavily on hand-crafted features and domain-specific knowledge (Cristea et al., 2016).
1https://multiconer.github.io
The MultiCoNER II shared task (Fetahu et al., 2023b) aims at building NER systems for 12 languages, namely English, Spanish, Hindi, Bangla, Chinese, Swedish, Farsi, French, Italian, Portuguese, Ukrainian and German. The task has 12 monolingual tracks and a multilingual one.
NER helps one easily identify key elements in a text, like athletes, politicians, human settlements, clothing pieces, medications and more. Extracting the main entities in a text helps sorting unstructured data and detecting important information, which is crucial if one has to deal with large datasets.
The datasets mainly contain sentences from three domains: Wikipedia, web questions and user queries which are usually short and low-context sentences (Malmasi et al. (2022a), Fetahu et al. (2023a)). Moreover, these short sentences usually contain semantically ambiguous and complex entities, which makes the problem more difficult. Usually, retrieving knowledge related to such ambiguous concepts in any form (like an article or results of a web search) is a definite method of understanding and disambiguating them. Thus, the ideal NER model would be capable of taking on hard samples if the option of additional context information was available.
The rest of the paper is organized as follows: section 2 briefly presents studies related to NER, either in a multilingual context or not, section 3
1107
presents the dataset, the required pre-processing and plausible methods for it, section 4 resumes the results of the conducted experiments, with their interpretations, followed by section 5 with the conclusions.

heading: 2 Related work
text: Currently, processing complex named entities is still a challenging NLP task. There is relatively little work on recognizing other types of entities than the traditional ones (persons, locations, organizations, date, currency). Complex NEs, like chemicals, ingredients, diseases or active substances are not simple nouns and are much more difficult to recognize (Mitrofan and Pais, 2022). They can take the form of any linguistic constituent and have a very different surface from than traditional NEs. Their ambiguity makes it challenging to recognize them. Additionally, more and more people share nowadays information about various topics online, making NER for these non-traditional entities more important in the context of data collected from social media, where people’s interests are directly expressed (Ashwini and Choi, 2014).
There has been made an attempt to investigate the ability of modern NER systems to generalize effectively over a variety of genres. This attempt also found out, as expected, that there is a strong correlation between the NER performance and the training corpus size, so by having a bigger corpus, the results may be more accurate (Augenstein et al., 2017). The job of handling NEs by extracting them from the text has been done by transformers. In the last few years, new technologies have appeared, including a Google research releasing mT5, their own version of transformer, which outperforms the previously released multilingual transformers (Xue et al., 2020).
Among those, BERT is one of the most powerful unsupervised models. A multilingual variant of it, trained over 100 languages and enhanced with context-awareness thanks to a CRF layer on top, has been leveraged before for such a task with promising results (Arkhipov et al., 2019). Lastly, the most successful approaches used in the previous MultiCoNER SemEval task were multilingual systems that revolved around improving upon the baseline model of XLM-RoBERTa, adapted to the multilingual track and then specialized for each monolingual track, as well as an enhanced BiLSTM network (Malmasi et al., 2022b).

heading: 3 Dataset and Methods
text: Although we explored a few options, we opted for the BERT transformer model for our approach. In this section, we present statistics over the dataset, as well as the steps we went through before choosing the BERT model and using the data for training.

heading: 3.1 Architecture
text: The architecture we used to train our model is rather simple and plays directly into BERT’s functionalities.
The diagram below showcases the basic flow and steps, while the details are detailed in the following sections.

heading: 3.2 Dataset
text: The dataset that we are using, MultiCoNER II, is a large multilingual dataset used for NER, that covers domains like: Wiki sentences, questions and search queries across 12 languages. This dataset contains 26M tokens and it is assembled from public resources. MultiCoNER II defines the following NER tag-set, with 6 classes:
• Location (LOC): Facility, OtherLOC, HumanSettlement, Station
• Creative Work (CW): VisualWork, MusicalWork, WrittenWork, ArtWork, Software, OtherCW
• Group (GRP): MusicalGRP, PublicCORP, PrivateCORP, OtherCORP, AerospaceManufacturer, SportsGRP, CarManufacturer, TechCORP, ORG
• Person (PER): Scientist, Artist, Athlete, Politician, Cleric, SportsManager, OtherPER
• Product (PROD): Clothing, Vehicle, Food, Drink, OtherPROD
• Medical (MED): Medication/Vaccine, MedicalProcedure, AnatomicalStructure, Symptom, Disease
Figure 2 presents a snippet with annotated entities from the dataset:
The dataset is split in three: (i) training dataset - used to train our model, (ii) development dataset - used for prediction in the practice phase and (iii) test dataset - used for prediction in evaluation phase. These three splits were made for each language. Table 1 shows some statistics of the dataset. For Latin languages, the number of instances is considerable higher over the three splits.

heading: 3.2.1 Preprocessing
text: We have concatenated the training data from all of the languages into a single CONLL file. Then, to make reading and processing a bit easier, we
have converted the data into CSV format, with the following structure (see Figure 3):
At this step, we took note of the number of 2,671,439 total entries. Spread between 67 NE tags, there was a clear discrepancy of distributions (highlighted in Figures 4 and 5).
Finally, we have grouped the entries by sentence number and have used this format (Figure 6) of the data going forward with the training.
This dataset had a final size of 166,413 entries, or better said sentences.

heading: 3.2.2 Preparation
text: Having processed our dataset, it was now time to prepare it for training. We started by having two maps ready:
• labels_to_ids which would associate each unique NE tag a unique number (having 67 total tags, we simply numbered them 0 to 66)
• ids_to_labels being the reverse map of the above
Then for each pair of (sentence, tags) in the dataset, we encode the sentence’s words using a tokenizer with a padding of 128 and convert the tags to their numeric form using our first mapping. The encoded words are then converted into tensors and each of them will be associated with the numeric labels which, similarly, are also converted into tensors. The padding values, as well as word pieces which are not in the first part of the word after tokenization, are attributed a custom value of -100. This can be better visualized in Figure 7.
Considering the final transformed model we ended up using, the tokenizer we applied for
this was, appropriately, the bert-base-uncased tokenizer.
The training set was turned into a DataLoader instance (from pytorch) and at this point was ready to be used.

heading: 3.3 Methods

heading: 3.3.1 Training
text: With a dataset of this size, we have run into difficulties trying to emulate the recommended baseline results with our own resources, as such we opted to try out different pretrained transformer models of small size to test which one would have the potential to be scalable within our limitations. Among the most popular and lightweight ones, we have decided on developing a model of our own based on the DistilBERT transformer.
Using it as a base, we have created a baseline English model that has been fine-tuned on the English training data and obtained decent enough results to begin building upon it. The results of this baseline are shown in Table 2. For the training parameters, we have used a learning rate of 1e-2, a batch size of 32, a number of epochs of 8 and a SGD (Stochastic Gradient Descent) optimizer.
With this experience, we went ahead and looked into what the BERT transformer would be capable of, by comparison. We have used the bert-baseuncased transformer model as a start and began transfer learning, this time, using the entire collection of training data for all of the languages. We were very pleased with the initial results of the model. This initial run used a learning rate of 1e05, a training batch size of 4 and a validation batch size of 2, just 1 epoch and the Adam optimizer.
Further testing used the same hyper-parameters, with the only difference being the number of epochs we trained the model for.

heading: 4 Results

heading: 4.1 Analysis
text: During the practice phase of the competition, we have submitted to each track a file that contains only the predicted tag for every token.
In Table 4 there is an example from the en_dev file, and below we can find the predicted tags that we must submit in the CodaLab competition:
O O O O O O O O O O O O B-CORP O B-CORP O
The weighted results that we have achieved with our model on the dev files can be seen in Table 5.
There are 2 scores that are noteworthy here, one regarding each token to its predicted tag (Table 6) and the other one regarding the tag predicted being in the correct tag-set. (Table 7)
We can observe that, compared to the prediction of the tags for each individual in all of the languages, the class in which each of the predicted tags can be found has increased scores. This means that, although it does not find the exact tag, it predicts another tag inside the same tag-set.

heading: 4.2 Evaluation
text: We were able to get results from all languages in the practice phase, however simulated errors were added in the datasets of the evaluation phase (except for DE test data) and our model could not
handle them properly. On a small scale (2-3 characters), we were able to resolve those problematic characters, but in languages that we were not familiar with, we had difficulty in detecting them.
Similarly to the practice phase, in Tables 8 and 9 are the results we have achieved with our model during the evaluation phase for the languages where we could successfully handle the input.
Looking at similar approaches from the previous year (SemEval-2022), we noticed a few papers that each add a specific component that elevate the results above the baseline:
• In Ma et al. (2022), before feeding the pretrained BERT model to build their NER system, they concatenated input text with entity information from the LUKE dictionary using string matching. Having participated only in the English track, they achieved an average F1-score of 0.7837, compared to our 0.61.
• In Boros et al. (2022), while using the same BERT encoder with a transformer layer and a CRF head for classification, they similarly preprocess the test text by adding to it a similar sentence from the training set chosen using SentenceBERT beforehand. They participated in all language tracks from last year with the best results in German (average F1-score of 0.7723 against our 0.55). The other shared languages between us are English (average F1score of 0.7196 against our 0.61) and Spanish (average F1-score of 0.6893 against our 0.54)
• In Pandey et al. (2022), they tested multiple classification heads apart from the CRF layer and found that for English the best approach was a linear layer (average F1-score of 0.7174 against our 0.61), while for other languages like Spanish they found the best approach to be pretraining BERT using the Whole Word Masking learning objective over Wikipedia data and the CRF layer (average F1-score of 0.612 against our 0.54).
Even though Italian and Swedish were not part of last year’s task, we understand from these papers that it is a difficult endeavor to have a single model capable of achieving great results across this spectrum of language. We limited ourselves by over-relying on a single model in this case.
As far as rankings are concerned, we have managed to get moderate results in the English track (we ranked 17th out of 34), while our results in the other tracks could be further improved in the future (overall third to last).

heading: 5 Conclusion
text: In this paper, we got the opportunity to explore a transformer model’s capabilities at dealing with NLP tasks - in this case complex NER - and how to handle task-specific input. More specifically, we put the classic BERT model to the test and found it to live up to its reputation of general-purpose transformer model by managing moderate results.
We have learned a lot about the workings of the transformer model and now have a better understanding of what tackling such a task entails with regards to approaches and resource management.
Among the things that could improve the results of this particular model, one important thing would surely be a more versatile module for handling input test data. Contrary to expectation, we should have put more focus on this part of the system. Apart from that, parallelization of the system could have potentially made it available to us to harness more powerful transformer models.

heading: Future Work
text: As far as rankings are concerned, we have managed to get moderate results in the English track (we ranked 17th out of 34), while our results in the other tracks could be further improved in the future (overall third to last)