LLM-RM at SemEval-2023 Task 2: Multilingual Complex NER using XLM-RoBERTa
Year: 2023

Abstract:
Named Entity Recognition(NER) is a task of recognizing entities at a token level in a sentence. This paper focuses on solving NER tasks in a multilingual setting for complex named entities.Our team, LLM-RM participated in the recently organized SemEval 2023 task, Task 2: MultiCoNER II,Multilingual Complex Named Entity Recognition. We approach the problem by leveraging cross-lingual representation provided by fine-tuning XLM-Roberta base model on datasets of all of the 12 languages provided Bangla, Chinese, English, Farsi, French, German, Hindi, Italian, Portuguese, Spanish, Swedish and Ukrainian.

heading: 1 Introduction
text: Named Entity Recognition(NER) is the task of recognizing entities (e.g., person, location, organization) in a piece of text.
Most of the NER datasets like CoNLL2003 (Erik F. Tjong and Meulder, 2003) are focussed on highresource languages like English and are specific to a given domain like news. The SemEval-2023 Task 2, Multilingual Complex Named Entity Recognition (MultiCoNER II) (Fetahu et al., 2023b) contains NER datasets curated for 12 languages which are low resource and from other domains as well. The task also provided a large number of 30 classes and simulated errors being added to the test set to make it more challenging. It contains the following languages: English, Spanish, Dutch, Russian, Turkish, Portugues, Korean, Farsi, German, Chinese, Hindi, and Bangla.
With the advent of Deep Learning based models, Transformer models like BERT, and bidirectional LSTM based ELMo became state of the art. BERT’s multilingual counterpart, mBERT became state of the art in multilingual NER tasks. Another model XLM-RoBERTa(XLM-R) (Alexis Conneau and Stoyanov, 2020) has shown to outperform mBERT on low resource languages on the NER
tasks. XLM-R is suitable for our task as it is pretrained for more than 100 languages including the 4 languages which we worked upon. Therefore, we have used XLM-RoBERTa-base transformer model and fine-tuned it on each of the given 12 languages.

heading: 1.1 The Challenge
text: We participated in the shared task named MultiCoNER II, which is part of Semeval 2023. The task consists of building a named entity recognition system in 12 languages. There are 2 tracks to the tasks, one is the monolingual track for each language and the other is a multilingual track consisting of all languages. The condition of the competition was that one can only use the prediction of the monolingual model in the monolingual track and cannot use it for the multilingual track
The task is a successor to the 2022 challenge of MultiCoNER shared task, (Malmasi et al., 2022b) where the key challenges were the following 1) syntactically complex entities like movies titles e.g Shawshank Redemption 2) these complex entities having low context and 3) long-tail entity distributions.
These challenges of NER for recognizing complex entities and in low-context situations were mentioned by Meng et al. (2021). The authors mention that for complex entities, some particular types (e.g creative works) can be linguistically complex. They can be complex noun phrases (Eternal Sunshine of the Spotless Mind), gerunds (Saving Private Ryan), infinitives (To Kill a Mockingbird), or full clauses (Mr Smith Goes to Washington). For long-tail entities, they mention that many domain entities have large long-tail distribution, with potentially millions of values (e.g., location names). It’s very hard to build representative training data, as it can only cover a portion of the potentially infinite possible values.
The current MultiCoNER II challenge, expands the previous challenge of 2022 with new tasks and
453
it emphasize the shortcomings of the current top models that are transformers based and depends on knowledge bases. It focusses on challenges like out of knowledge-base entities and noisy scenarios like the presence of spelling mistakes and typos.

heading: 2 Related Work
text: The earliest use of the neural networks to solve NER was by (Hammerton, 2003) who attempted to use an LSTM network to solve the problem. Recent transformer networks like BERT (Jacob Devlin and Toutanova, 2018) and ELMo (Matthew E. Peters and Zettlemoyer, 2018) have further led to the state of the art results in the NER task for English language.
CRF layer (Sutton and McCallum, 2012) was also proposed to be used as a final layer for token classification.
To expand NER in a multilingual setting, (Fetahu et al., 2021) introduced a multilingual and code-mixed dataset concerning the search domain. They also used an XLM-RoBERTa model combined with gazetteers to build their multilingual NER classification system.
In a paper for the first edition of this task in 2022, XLM-RoBERTa has been applied to Hindi and English dataset and have shown to perform better than mBERT with a similar set of languages as in this current task (Sumit Singh and Tiwary, 2022).

heading: 3 Data
text: The dataset was first released in the Semeval task in 2022, called MultiCoNER dataset (Malmasi et al., 2022a). In the 2nd edition of the task, The organisers provided a new dataset called MultiCoNER v2 (Fetahu et al., 2023a) comprising of individual language datasets in 12 languages.
Table 1 contains the number of sentences in the training, development and test datasets per language in MulticoNER v2 dataset.
The test dataset is used for the final evaluation of the leaderboard and is further split into corrupted and non-corrupted sets. It is to be noted that the uncorrupted test set size is quite large compared to the training set for all the languages.
Table 2 contains the list of 30 entities present across all datasets and their grouping into the corresponding entity types.

heading: 4 Methodology
text: We utilised XLM-RoBERTa model for all 12 languages we participated in. XLM-RoBERTa is a massive Transformer trained on 100+ languages on 2TBs of CommonCrawl data.
Figure 1 describes the architecture of transformer encoder models like RoBERTa. It is trained with a multilingual MLM objective using only monolingual data. XLM-RoBERTa has already been shown to outperform mBERT on cross-lingual classification by upto 23 percent accuracy on lowresource languages. It also gave competitive results with respect to the state of the art monolingual models.

heading: 4.1 Experimental Set up
text: We have used the PyTorch framework and HuggingFace’s Tranformers library for our system.The training is done on a single GPU Nvidia Titan 2080 machine.
We used the XLM-RoBERTa-base model and fine-tuned it on each of the language datasets. We train the model in a batch size of 4 on the training dataset. A dropout of 0.2 is applied between the last hidden layer and the output layer to prevent the model from overfitting. We used a learning rate of 2e-5. For the optimizer, we used AdamW (Loshchilov and Hutter, 2019) optimizer with an epsilon of 1e-08. AdamW is a stochastic optimization method and it modifies the implementation of weight decay in Adam, by separating weight decay from the gradient update. The max length of the sequence used is 16. We trained the model on 15 epochs and select the best one based on f1-scores on the test set.
For token classification at the end, we used a linear layer as the last layer to classify into the given 30 named entities.
The final model is based on the checkpoints of the model which are selected based on best F1scores on the development set.

heading: 5 Results
text: Table 3 shows the macro average scores of precision, recall and F-1.
From Table 4, we observe that the entities belonging to the Creative Work category consistently have the highest F1-score across all 4 languages. Also, it can be noted that the entities belonging to Person and Product group have the lowest F1scores for the model.
Table 5 shows the macro average F1-scores of the final test set used for leaderboard
From table 3 and table 5, we observe that the performance for Hindi drops most from development to test set, while for English and Spanish, it just drops slightly.
Also, we couldn’t submit results for languages other than Hindi,English,German and Spanish by the time the competition ended, therefore we only have F1,corrupted f1 and uncorrupted F1 scores for these languages for the test set.

heading: 6 Conclusion
text: In this paper, we presented using XLM-RoBERTabase to solve the shared task of MultiCoNER.
Future work can include exploring more recent transformer-based models like XLM-V with very large vocabularies. Also, data augmentation techniques like entity replacement can be tried.

heading: Future Work
text: Future work can include exploring more recent transformer-based models like XLM-V with very large vocabularies. Also, data augmentation techniques like entity replacement can be tried