IXA at SemEval-2023 Task 2: Baseline Xlm-Roberta-base Approach
Year: 2023

Abstract:
IXA proposes a Sequence labeling fine-tune approach, which consists of a lightweight baseline (10e), the system takes advantage of transfer learning from pre-trained Named Entity Recognition and cross-lingual knowledge from the LM checkpoint. This technique obtains a drastic reduction in the effective training costs that works as a perfect baseline, future improvements in the baseline approach could fit: 1) Domain adequation, 2) Data augmentation, and 3) Intermediate task learning. Practical Information The system is composed of two main stages, first of all, fine-tuning which consists in learning IOB sequences via sequence classification task, and then the inference phase when unannotated examples are given to guess the correct IOB sequence. We learn the correct way to label sequences using real examples from the training set and backpropagate the labeling errors over the development set compared to the intended gold standard. At this point, we are taking advantage of already pre-trained checkpoints over MLM and Sequence Classification tasks. Figure 1: Trainning Schema. Finally, we infer the proper IOB sequences for raw examples, those inferences are led by Language Model expertise on previous sequence classification plus the fine-tuning on MulticoNER this property is called transfer learning, additionally the Masked Language Modelling allows the unseen token representation into the already known word space via substrings in this case, this property is called cross linguality. Figure 2: Inference Schema.

heading: 1 Introduction
text: The tag set of MultiCoNER is a fine-grained taxonomy 3 with 36 tags divided into 6 main groups. Further relevant information related to SemEval shared tasks is provided in guidelines (Fetahu et al., 2023).
In addition to multilingualism, there were two key challenges in this year’s task: (1) a fine-grained entity taxonomy which mainly penalizes the precision of systems as the ability to guess the correct tag, and (2) simulated errors added to the test set to make the task more realistic and difficult which mainly penalizes the recall of systems as the ability to guess the correct chunk of text to tag.

heading: 2 Related Work
text: SemEval-2023 Task 2 follows the multilingual NER task started in 2022 (Malmasi et al., 2022b). critical challenges in the 2022 datasets were dealing with complex entities with limited context (Malmasi et al., 2022a). This is commonly understood as a low-resourced specific domain problem e.g "Medical Entity Recognition", main
379
issues come from the fact that only a few of the total examples are annotated and the provided contexts show specific lexical usually unknown for language models e.g "in particular he worked on the laser photocoagulation of threshold retinopathy of prematurity".
The challenges of Sequence Classification for recognizing complex entities in low-context conditions was recently outlined by Meng et al. (2021). Other work has extended the content to multilingual and code-mixed settings (Fetahu et al., 2021). In our experience, the multilingual constraints force the necessity of data to improve the learning results, here we identify three main paths: 1) Domain adequation, which consists in learning MLM task with the raw texts provided in the task, 2) data augmentation, which is generating synthetic data with a sustainable lexical difference according to real examples but maintaining the possibility to align the annotated IOB chains, and 3) Intermediate task learning, which feeds Language Model with more expertise via Silver Dataset learning (Similar tasks in other formats), this is usually pursued in more generic fine tune tasks such as Natural Language Inference.

heading: 3 Data
text: As we can appreciate we are provided a finegrained taxonomy with 36 labels divided into 6 main groups. The data collection methods used to compile the dataset used in MultiCoNER will be described in a paper to be published shortly (Malmasi et al., 2022a). We are provided a dataset divided into train, dev, and test sets, in this case,
mainly unannotated corpora (test set) as shown in table 1.

heading: 4 Results
text: We pursued 12 experiments one per language, each xlm-roberta-base (Conneau et al., 2019) model was trained with 32 batch sizes for 10 epochs. The training process took on average around 25 minutes using NVIDIA Tesla V100 GPU. We took around 13h using 400W that is 5.20 KW which could be understood as 7,84 CO2e (lbs) generated by the experimentation.
We provide two result tables to describe the proposed system performance, development evaluation shown in table 2 was performed via seqeval framework (Nakayama, 2018), this evaluates the performance of chunking tasks. And test evaluation is shown in table 3. The ranking is computed using Macro averaged F1 scores. In the held-out test
set, we had a noisy subset for 8 languages where the sentences were corrupted with noise either on context tokens or entity tokens. The test sets were corrupted for EN, ZH, IT, ES, DE, FR, PT, SV. In this case fine-grained metrics were assessed.

heading: 5 Conclusion
text: These brief experiments are considered baseline approaches to the task. Those techniques applied in various domains/tasks show that we can effectively approach the baseline system as a sequence labeling fine-tune over the dataset. Additionally, future improvements in the baseline approach could fit: 1) Domain adequation, 2) Data augmentation, and 3) Intermediate task learning.

heading: Future Work
text: Additionally, future improvements in the baseline approach could fit: 1) Domain adequation, 2) Data augmentation, and 3) Intermediate task learning