muNERa at WojoodNER 2024: Multi-tasking NER Approach (2024)

Abstract: This paper presents our system “muNERa”, submitted to the WojoodNER 2024 shared task at the second ArabicNLP conference. We participated in two subtasks, the flat and nested fine-grained NER sub-tasks (1 and 2). muNERa achieved first place in the nested NER sub-task and second place in the flat NER subtask. The system is based on the TANL framework (Paolini et al., 2021), by using a sequenceto-sequence structured language translation approach to model both tasks. We utilize the pretrained AraT5v2-base model as the base model for the TANL framework. The best-performing muNERa model achieves 91.07% and 90.26% for the F-1 scores on the test sets for the nested and flat subtasks, respectively.

heading: 1 Introduction
text: The abundance of written texts in the digital age has increased the importance of identifying key pieces of information, known as named entities, within the text. This process, called Named Entity Recognition (NER), involves identifying and classifying entities into pre-defined categories, such as names of persons, organizations, locations, dates, and more, depending on the application. The ability to accurately identify these entities is crucial for numerous real-time applications, including biomedical and clinical studies (Liu et al., 2022), business (Shah et al., 2023), and law (Kalamkar et al., 2022). Moreover, NER complements other NLP tasks, including Question Answering (Guven and Unalir, 2021), Machine Translation (Mota et al., 2022), and Coreference Resolution (Wang and El-Gohary, 2023).
This paper presents our system, muNERa, which participated in the WojoodNER 2024 shared task at the second ArabicNLP conference (Jarrar et al., 2024). The WojoodNER shared task, now in its second year, includes two subtasks: flat and nested fine-grained NER. Also, there is a third “open track”
subtask in which we did not participate. WojoodNER aims to advance the state-of-the-art in Arabic NER by providing a large-scale, fine-grained, and annotated dataset and a competitive platform for researchers to evaluate their models. The main difference between WojoodNER 2024 and 2023 is the introduction of WojoodFine with finer-grained sub-tags and sub-sub-tags (Liqreina et al., 2023). Also, there has been a major revision of the tagging schemes.
Typically, NER is modeled as a sequence classification problem such that each token is assigned a tag using a sequence tagging scheme such as IOB2 (Tjong Kim Sang and Veenstra, 1999). In muNERa, we model both the nested and flat NER tasks as a sequence-to-sequence translation utilizing the Translation between Augmented Natural Languages (TANL) framework (Paolini et al., 2021). Notably, all participating teams in the Wojood 2023 shared task (Jarrar et al., 2023) did not utilize sequence-to-sequence (encoder-decoder) architecture, their work employed BERT pre-trained language models (encoder-only) in different ways (Ehsan et al., 2023; Laouirine et al., 2023; El Mahdaouy et al., 2023; Li et al., 2023; Elkordi et al., 2023).

heading: 2 Work on WojoodNER 2023
text: WojoodNER-2023, the first Arabic NER shared task (Jarrar et al., 2023), includes FlatNER and NestedNER subtasks with AraBERTv2 as the baseline (Antoun et al., 2020). Notably, most participating teams utilized BERT pre-trained language models, with the performance of the encoderdecoder T5 model remaining unexplored (Raffel et al., 2023). Seven studies employed AraBERT (Antoun et al., 2020), and it is remarkable that AraBERT consistently outperformed other transformers: LIPN (El Elkhbir et al., 2023) achieved first place in FlatNER, ELYADATA (Laouirine
858
et al., 2023) excelled in NestedNER. El-Kawaref (Elkaref and Elkaref, 2023) utilized StagedNER, while UM6P & UL (El Mahdaouy et al., 2023) adopted multi-task learning. Additionally, Lotus (Li et al., 2023) explored multi-task learning with XLM-R, AraBERT, and MARBERT, favoring XLM-R. AlexU-AIC (Elkordi et al., 2023) employed a machine reading comprehension-based approach, leveraging various PLMs, while Alex-U 2023 NLP (Hussein et al., 2023) introduced AraBINDER, utilizing BERT models.
In our work, we explore a new method utilizing the T5 encoder-decoder architecture instead of the BERT encoder architecture.
منجانبھرحبالسیدیوسفحبشالممثلالمقیملمؤسسةالتمویلالدولیةفيفلسطین PERSOCCGPE
COUNTRY
Parent Sub-tag
Flat NER
Nested NER

heading: 3 Data
text: The Wojood Corpus is a dataset for Arabic NER dataset containing approximately 550,000 tokens of Modern Standard Arabic and dialects. This dataset is designed specifically to identify nested entities, where entities are embedded within other entities. The Wojood Corpus consists of two versions for Arabic nested NER: the original and the WojoodFine Corpus (Jarrar et al., 2022; Liqreina et al., 2023).

heading: 3.1 WojoodFine Corpus
text: WojoodFine (Liqreina et al., 2023) is a comprehensive Arabic corpus for nested NER focusing on fine-grained entity types across various domains. This refined extension of the original Wojood Corpus (Jarrar et al., 2022) contains tokens annotated across various entity types, including geopolitical entities (GPE), locations (LOC), organizations (ORG), and facilities (FAC). Each entity type contains sub-types from 31 sub-types following the LDC’s ACE guidelines (Walker et al., 2005). This dataset features 47.6K fine-grained and nested annotations, allowing for more detailed entity recognition than provided by standard NER tasks. Table 5
in Appendix A details the frequencies of the 53 NER tags present in the training and development data splits.

heading: 3.2 Data Preprocessing
text: As detailed in section 4, we adapted the WojoodFine Corpus to meet the input requirements of the TANL framework (Paolini et al., 2021) to use it for modeling effectively. The preprocessing steps involve extracting hierarchical tags (parent, subtag, sub-subtag) and their spans using the BIO scheme. Then, each token and its corresponding labels are reformatted to align with the TANL framework’s specifications. For example, the sentence “¨ ¨ As ¤dnm Y T AFC . yWsl ” will be, before preprocessing (using the BIO scheme),: [(T AFC, O), (Y , O), ( ¤dnm , BOCC), (¨ As , I-OCC), (¨ , I-OCC), ( yWsl , B-GPE, I-OCC), (., O)] ; and after preprocessing: tokens: [,¨ ,¨ As , ¤dnm ,Y ,T AFC . , yWsl ], entities: [type: GPE, start: 5, end: 6, type: OCC, start: 2, end: 6].

heading: 3.3 Challenges
text: This subsection discusses some challenges we faced while working on the WojoodFine Corpus.
Complexity of the Tagging System: The tagging architecture employed within the WojoodFine Corpus introduces a three-level nested tagging scheme that significantly complicates the annotation and recognition processes. In this scheme, a single token can be associated with many different parent tags (in the case of nested tagging). Each parent tag may have a sub-tag as shown in Figure 1, and each sub-tag can include up to two sub-sub tags. This results in a multi-layered tagging structure where a single token can be annotated with many tags across the three levels. In fact, in the WojoodFine Corpus, some tokens have up to five different parent tags and eight subtags.
This complexity increases the difficulty of accurately tagging the data and poses substantial challenges for the NER models in learning and predicting such a diverse range of tag combinations effectively.
Inconsistency in Tag Distribution: Some tags, such as the sub-tag “ENT”, were present in the development set but not in the training set. This inconsistency can create significant challenges during model training, as the model may not learn
how to recognize entities that only appear in the development or test set but not in the training set.
Sparse Examples for Certain Tags: The dataset shows that several tags are rarely represented. For example, the parent tag “UNIT” only appears six times in the training set. Similarly, the sub-tag “LAND” appears only once each. Another example is the tag “Path”, which differs from “PATH” due to case sensitivity and has only one example in the training set. This issue also extends to sub-sub tags like “ENT” and “PLANT”, which have only one and three examples, respectively. Having a few number of examples for some tags makes it difficult for models to recognize them accurately.

heading: 4 Methodology
text: We utilize the TANL framework (Paolini et al., 2021) to train our models for both flat and nested NER. The TANL framework is designed to address structured prediction tasks in language processing. These tasks include joint entity and relation extraction, relation classification, semantic role labeling, event extraction, coreference resolution, dialogue state tracking, and nested named entity recognition (Paolini et al., 2021).
One key advantage of the TANL framework is its ability to incorporate the semantic understanding of labels during model training. Unlike traditional task-specific classifiers, which typically train without explicit knowledge of label semantics, TANL leverages pre-trained models’ understanding of entity semantics (e.g., Person, Location). In the TANL framework, depending on the task, both the input and output are structured in augmented natural languages. In our case, we created the NER structure task in TANL similar to the Adverse Drug Effect (ADE) dataset (Gurulingappa et al., 2012) structure task, as both datasets feature nested NER entities. The task aims to extract entities and their spans from a given sentence. These entities are
enclosed within the special tokens [ ], and each entity is followed by its type and separated by |, e.g., [ token | entity type ]. Entity types are represented in natural language words—such as person or location—rather than abbreviations like PER or LOC to leverage the model’s semantic knowledge of these words. For WojoodNER labels, we manually created a dictionary mapping the entity tags (e.g., PER) to their natural language words (e.g., person) based on the description of the tags in the WojoodFine paper (Liqreina et al., 2023). Some entity tags are already in a natural language format, such as “PRODUCT” and “SPORT”, while others require extracting the appropriate natural language terms from their descriptions as shown in Table 1.
For Nested entities, the task allows the representing of entity hierarchies, for example [ token [ token | entity type1 ] | entity type2 ]. The sub-tag and sub-sub-tags were treated the same as nested parent tags. In other words, a span with multiple tags (parents and subtags) will be treated as the whole span is nested within itself using the other tags. Figure 2 illustrates how the Wojood dataset is formatted as input and output using the TANL framework.
• For flat: entities §wk T ¤ and Hy¶C ryb Tn§d T§dl tagged as “geographical entity” and “occupation”, respectively.
• For flat with sub-tags: entity §wk T ¤ has
a sub-tag “country”.
• For Nested: entity ryb Tn§d T§dl Hy¶C has multiple parent tags, including T§dl ryb Tn§d and ryb Tn§d tagged as “organization” and “geographical entity”, respectively.
• For Nested with sub-tags: entities Tn§d ryb and §wk T ¤ have sub-tags “town” and “country”, respectively.
After generating the output, misspellings of the tokens in the original text or minor format errors can occur due to the nature of generative models. To address this issue, TANL’s decoding process first cleans the output by removing special tokens and discarding invalid formats, ensuring only valid entity types are considered. To further enhance robustness, TANL utilizes the dynamic programming (DP) based Needleman-Wunsch alignment algorithm (Needleman and Wunsch, 1970), which aligns input and cleaned output tokens at the token level. For instance, it can correctly align a misspelled generated token (e.g., ryb Tn§d ) with its correct form in the input (e.g., ryb Tn§d ), ensuring accurate entity identification. We noticed that most of the invalid outputs are:
• Invalid format, such as missing open bracket “[” or the closed bracket “]” tokens. For example, the generated output “Yl T ¯ [  rm [ [ Tylyql Tn§d | town ] | geographical entity ] | facility ] | building or grounds ] ...”’ has a missing open bracket “[” token. In this case, the last entity “building or grounds” is discarded before the alignment process.
• Missing the end-of-sentence period “.”, and repeating some tokens, such as “ y T§dl Hy¶C T AFC y TqWn ¨ ©rks` A l y TqWn ¨ ©rks` A l ” instead of the original text “Hy¶C T AFC ¨ ©rks` A l y T§dl y TqWn ”.
• Additionally, some invalid inputs include generating other variations of Arabic dialects. For example, the original text “¨ wJ : E¤ry . ¢ A lk ¨ wJ” was reconstructed as “.¢ A lk ¨ wJ ¨ wJ : E¤ry ”.
In our case, although there were a few examples where the entities in invalid format text, such as
missing the open “[” or closed “]” bracket, were discarded, all of the invalid reconstructed examples in the development and test sets were successfully aligned.
We use the AraT5v2 model (Elmadany et al., 2023) as the base for the TANL framework. AraT5v2 is a pre-trained Arabic model based on the T5 (Text-to-Text Transfer Transformer model) encoder-decoder transformer architecture (Raffel et al., 2020). We utilize TANL based on AraT5v2 as a multi-tasking span detector and entity recognition such that it outputs the structured predictions of spans and their entity types. We use two distinct TANL models: one for flat NER and one for nested NER. We also use fastText (FT) classifier (Joulin et al., 2016) as a secondary tagger such that we first use TANL to detect spans and assign level-1 tags (parent tags), then we tag the detected span with level 2 and 3 using the FT classifier.
Evaluation metrics: We follow the WojoodNER shared task evaluation criteria and report the microaverage scores for Precision (P), Recall (R), and F-1 (F). Additionally, we report the F-1 score for span detection, where every span in the evaluation set is considered positive regardless of its type.

heading: 5 Experiments and Results
text: We use the TANL model (Paolini et al., 2021), based on AraT5v2 (Elmadany et al., 2023) as the primary classifier, and fastText (FT) (Joulin et al., 2016) trained on spans from the training set as a secondary classifier. For FT, we use the preArabic trained vectors fasttext-ar-vectors (Grave et al., 2018), which is available on HuggingFace 1. For TANL, we set the values for hyperparameters as follows:
Hyperparameter Value
Learning rate (LR) 5e− 4 Beam Size 8
Number of epochs 5
Train Batch size 4
Max sequence length 768
To train the TANL model, we use Microsoft Azure ML service with the NC A100 v4 series, powered by Nvidia A100 GPUs with 80 GB memory. For inference, we use Google Colab Pro+ with one Nvidia A100 GPU.
1https://huggingface.co/facebook/ fasttext-ar-vectors
Table 2 shows the results on the validation set for different models. TANLp is a TANL model trained only on level-1 tags (parent tags), and TANLs is trained on all levels (i.e., parent tags, sub-tags, and sub-sub-tags). (+ FT) indicates additional tagging of the detected spans using the FT model with the remaining levels (i.e., sub-tags and sub-sub-tags).
The results indicate that the best model on both tasks (i.e., nested and flat) is TANLs, which tags all levels without additional tagging using the secondary classifier FT. We observe that all TANLbased models perform well in span detection and recognition. Additionally, when using fastText as a secondary tagger with TANLp, the results indicate that the performance is close to TANLs for nested (less than 1% difference in F-1), but not the case with the flat subtask. This may be due to overpredicting entity types by fastText, as reflected in the precision and recall scores.
For TANL with the simpler tagging scheme (i.e., TANLp), the model performs better than the model with subtags in span detection for the flat subtask. However, for the nested subtask, the model trained on subtags performs better than the model trained on only parent tags, TANLp.

heading: 5.1 System Submissions
text: Based on the validation set results shown in Table 2, we select TANLs as the best model for both tasks: nested and flat. We report the performance on the blind test set in Table 3, showing the rank of our system among other submissions to the two subtasks.

heading: 6 Conclusions
text: This paper presents the results of “muNERa” on the WojoodNER 2024 shared task. The muNERa system demonstrated outstanding performance in the WojoodNER 2024 shared task, securing first place in the Nested NER sub-task and second place in the Flat NER sub-task. The system’s success is primarily attributed to the innovative application of the TANL framework (Paolini et al., 2021), combined with the AraT5v2-base model (Elmadany et al., 2023). These methodologies allowed muNERa to achieve remarkable F-1 scores of 91.07% for Nested NER and 90.26% for Flat NER.
Our system shows that the encoder-decoder architecture tackles the complexity of the WojoodFine Corpus annotation scheme, effectively modeling the multi-layered tagging scheme and the challenge of the nested entities structure. Despite the challenges posed by inconsistencies and sparse examples in the dataset, muNERa consistently delivered high-precision and recall metrics.
For future work, there are several directions. One is integrating the hierarchical structure of the 3-level tagging scheme (i.e., tags, sub-tags, and sub-sub-tags) into the model, potentially improving performance. An example is experimenting with different structures for the TANL format that incorporate the hierarchical tagging scheme. Additionally, employing more advanced secondary taggers, like BERT, could improve the performance further than our secondary tagger, fastText. Finally, addressing the computational limitations and experimenting with a broader range of hyperparameters will also be crucial for future advancements.

heading: 7 Limitations
text: There are some limitations regarding the design of muNERa for the WojoodNER 2024 shared task.
• The system does not consider the hierarchical structure of the 3-level tagging scheme. This can be a future direction, such that the model considers sub-tags and sub-sub-tags to be associated with certain main tags.
• We use two distinct TANL models, one for each sub-task (i.e., nested and flat). Training a single model on both tasks, allowing the task to be prompted in the input, is a future research direction.
• We have shown in subsection 3.3 some challenges in the WojoodFine dataset. One is the class imbalance challenge. Also, we have shown that some tags were present in the development and test set but not in the training set. Due to computational and time constraints, we haven’t experimented with data resampling techniques.
• We have used English natural language words for entities as described in section 4. We have not tried to use Arabic words. We believe this would be a good direction to enhance the model, especially since our base model, AraT5v2, was pre-trained mainly on Arabic datasets.
• We have used fastText as a secondary tagger with TANL as the primary one. We did not use a more advanced secondary tagger, such as BERT. This can be a future direction where more advanced models can be used as secondary taggers.
• Due to computational limits, we have not experimented with a wider range of hyperparameters, such as the number of epochs or beam size for TANL.

heading: Acknowledgments
text: The authors would like to thank the anonymous reviewers for their valuable comments and feedback. We also thank SDAIA for providing access to Microsoft Azure Service to train the models and KSAA for providing Google Colab Pro+ accounts to evaluate and test the models used in this
paper. Finally, we would like to thank the WojoojdNER 2024 shared task organizers for organizing the shared task and for their prompt and helpful responses to our inquiries.

heading: A Detailed results
text: Table 4 shows the details of results for the best model for nested NER in section 5 (TANLs). Table 5 shows the distribution of entity classes in the train and development sets.

heading: Future Work
text: This can be a future direction, such that the model considers sub-tags and sub-sub-tags to be associated with certain main tags. • We use two distinct TANL models, one for each sub-task (i. e. , nested and flat). Training a single model on both tasks, allowing the task to be prompted in the input, is a future research direction. • We have shown in subsection 3. 3 some challenges in the WojoodFine dataset. One is the class imbalance challenge. Also, we have shown that some tags were present in the development and test set but not in the training set. Due to computational and time constraints, we haven’t experimented with data resampling techniques. • We have used English natural language words for entities as described in section 4. We have not tried to use Arabic words. We believe this would be a good direction to enhance the model, especially since our base model, AraT5v2, was pre-trained mainly on Arabic datasets. • We have used fastText as a secondary tagger with TANL as the primary one. We did not use a more advanced secondary tagger, such as BERT. This can be a future direction where more advanced models can be used as secondary taggers. • Due to computational limits, we have not experimented with a wider range of hyperparameters, such as the number of epochs or beam size for TANL