Addax at WojoodNER 2024: Attention-Based Dual-Channel Neural Network for Arabic Named Entity Recognition (2024)

Abstract: Named Entity Recognition (NER) is a fundamental task in Natural Language Processing that focuses on extracting entities such as names of people, organizations, locations, and dates from text. Despite significant advancements due to deep learning and transformer architectures, NER still faces challenges, particularly in low-resource languages like Arabic. This paper presents a BERT-based NER system that utilizes a two-channel parallel hybrid neural network with an attention mechanism specifically designed for the NER Shared Task 2024. For the WojoodNER 2024 Shared Task, our approach ranked second by scoring 90.13% in micro-F1 on the test set. The results demonstrate the effectiveness of combining advanced neural network architectures with contextualized word embeddings in improving NER performance for Arabic. Code is available at https://github.com/issam-yahya/Addax.

heading: 1 Introduction
text: Named Entity Recognition (NER) is one of the main tasks of Natural Language Processing (NLP) and plays a fundamental role in different NLP applications, such as machine translation, information retrieval, and question-answering (Alami et al., 2023). NER involves extracting entities from unstructured text, including people’s names, organizations’ names, locations, and dates. These entities can be categorized as flat or nested based on their structure. Flat entities consist of a single non-overlapping span of text representing a single distinct entity. Nested entities, on the other hand, include entities that are embedded within other entities.
In this paper, we present our participating system for the second edition of the flat subtask of NER Shared Task 2024 (Jarrar et al., 2024), building on the foundations laid by the first edition (Jarrar et al.,
*These authors contributed equally to this work.
2023). Our system is a BERT-based model that utilizes a two-channel parallel hybrid neural network with an attention mechanism. The best results were achieved using AraBERT (Antoun et al., 2020). We evaluated our system’s performance using the micro-average F1 score, Recall, and Precision. On the test set, it achieved a micro-F1 score of 90.13% and secured the second position.

heading: 2 Related Work
text: Previous studies in Named Entity Recognition (NER) have employed diverse methodologies, including knowledge-based approaches, machine learning techniques, and deep learning methods. Knowledge-based approaches were among the earliest strategies used in NER. These methods relied on hand-crafted rules (Hanisch et al., 2005; Shaalan and Raza, 2007) and on language-specific knowledge such as lexical markers (Zhang and Elhadad, 2013) and entity dictionaries (Etzioni et al., 2005). With the rise of machine learning, studies shifted from hand-crafted rules to various machine learning techniques such as Support Vector Machines (SVM) (McNamee and Mayfield, 2002), Hidden Markov Models (HMM) (Morwal et al., 2012), and Conditional Random Fields (CRF) (McCallum and Li, 2003). Recently, significant progress in NER has been driven by advancements in deep learning (Li et al., 2022). Various deep learning architectures have been successfully incorporated into NER, including Convolutional Neural Networks (CNNs) (Gui et al., 2019), Recurrent Neural Networks (RNNs) (Lample et al., 2016), and Transformers (Labusch et al., 2019; Yan et al., 2019).
Recent models leveraging transformer architectures, such as BERT (Devlin et al., 2018) and its variants, have set new benchmarks in NER performance by capturing more contextual nuances and complex language structures than traditional models. Additionally, improvements in pretrained
867
language models (Devlin et al., 2018; Antoun et al., 2020) have enabled the adaptation of NER systems developed with fewer annotated data to new languages and domains, improving their accuracy and performance. However, NER still faces many challenges, especially in low-resource languages such as Arabic (Darwish et al., 2021).
Most Arabic NER research has focused on flat entities, primarily targeting a few coarse-grained entity types, such as person, organization, and location. Although coarse-grained NER is useful across various domains and serves as a foundational approach, it falls short for tasks that demand a more detailed understanding of named entities (Ling and Weld, 2012). In this context, WojoodFine (Liqreina et al., 2023) was introduced to address the scarcity of fine-grained Arabic NER corpora. It is an extension of Wojood (Jarrar et al., 2022) that introduced finer-grained sub-types. Wojood initially included 21 entity types and was primarily gathered from Modern Standard Arabic (MSA) articles, which constituted the majority of the data, with a smaller portion collected from social media in Palestinian and Lebanese dialects. WojoodFine enhanced the original Wojood corpus by introducing fine-grained entities to four entity types: Geopolitical Entity (GPE), Organization (ORG), Location (LOC), and Facility (FAC).

heading: 3 Data
text: The Wojood Fine corpus extends annotation of the Wojood original corpus (Jarrar et al., 2022) by including fine-grained annotations for named-entity subtypes. It contains 550K tokens and was manually annotated with 21 entity types. It is worth mentioning that approximately 80% of Wojood originates from Modern Standard Arabic (MSA) articles, but about 10% comes from content on social media sites that were written in Palestinian or Lebanese dialects via the Curras and Baladi corpora (Al-Haff et al., 2022). Additionally, Wojood has also nested named entities, though some of them could refer to more than one type of entity like ‘Organization’, which may interfere with information retrieval tasks further downstream. In response, WojoodFine provides subtypes for four main categories of entities: GPE, ORG, LOC, and FAC.

heading: 3.1 Main Entity Types
text: Table 1 provides an overview of the frequency of these four main entity types in both Wojood and WojoodFine.
The development of WojoodFine was informed by LDC’s ACE 2008 annotation guidelines for Arabic entities v7.4.2.

heading: 3.2 Sub-types
text: The sub-types introduced in WojoodFine provide a more granular level of annotation for GPE, ORG, LOC, and FAC entities, resulting in the addition of 31 new entity sub-types. The frequency of each subtype in the WojoodFine corpus is shown in Figure 1. WojoodFine is available as a RESTful web service, and both the data and source code are publicly accessible 1

heading: 4 System Overview

heading: 4.1 Data Pre-processing
text: Existing NER models use nested tagging formats like BIO, assigning tags to both primary entity types and their sub-types (El Mekki et al., 2022)
1https://github.com/SinaLab/ArabicNER
(e.g., B-GPE, I-GPE for Geopolitical Entities and B-CAMP, B-CLUSTER for sub-types). While informative, this method is complex due to the need to predict many independent tags. To address this challenge, we used combined tagging to classify entities. In this approach, we combined the main entity type and its sub-types into a single category. For example, the entity "Palestine" with a main type of B-GPE and a sub-type of B-COUNTRY, will be classified as "B-GPE+B-COUNTRY".
Combined tags follow the BIO scheme for entity boundaries, with "I-" and "B-" indicating the inside and beginning of entities, respectively. This approach simplifies training by focusing on a single combined tag per entity, integrating both main and sub-type information. It reduces complexity despite a slight reduction in granularity and potential challenges with complex nested entities.

heading: 4.2 Model Architecture

heading: 4.2.1 Architecture Overview
text: Our model, as shown in Figure 2, utilizes a twochannel parallel hybrid neural network with an attention mechanism (Chen et al., 2023) to address the unique characteristics of the Arabic language. Each Arabic word possesses both local properties and global properties, necessitating a specialized approach for effective processing. To achieve this,
our model uses BERT embeddings (Devlin et al., 2018) to generate contextualized word representations and utilizes two channels, each with a specific purpose. The first channel focuses on local feature extraction using CNNs, while the second channel captures long-range dependencies using RNNs, enabling the model to understand complex linguistic concepts within sentences.

heading: 4.2.2 Convolutional Layers Channel
text: The first channel of our model uses 3 stacked Conv1D layers to extract local patterns and features from the embeddings. Max pooling is applied to reduce feature map dimensions, allowing convolutional filters to focus on significant details and further preventing overfitting. Dropout regularization is incorporated to prevent overfitting by randomly nullifying some input units during training, thereby enhancing generalization. An ablation study was conducted, as shown in Table 4, comparing configurations with 1, 3, and 4 convolutional heads to assess the impact of the number of heads on performance.

heading: 4.2.3 Bidirectional GRU Channel
text: The second channel uses Bidirectional GRU layers to account for long-range dependencies in the input sequences, enhancing the model’s ability to recognize entities. This channel captures the sequence context bidirectionally, allowing the model to understand the context of each word within a sentence more effectively. We performed an analysis comparing the use of Bidirectional GRU with Bidirectional LSTM layers to evaluate their effectiveness, detailed in Table 2.

heading: 4.2.4 Attention Mechanism
text: The attention mechanism (Vaswani et al., 2017) plays a role in our architecture by weighing different parts of the input sequence, concentrating on the most relevant features for prediction. This mechanism significantly enhances the model’s performance by allowing it to focus on the most critical parts of the input data.
Moreover, input normalization stabilizes the training process, and additional dropout layers ensure robust classification without overfitting. The integration of the attention mechanism consistently improved results across various configurations, as shown in Table 2.

heading: 4.2.5 Attention-Based Dual-Channel Neural Network Algorithm
text: Algorithm 1 presents the detailed process flow of the system. The input sequences are initially processed using BERT, followed by a dropout layer to reduce overfitting. In the local feature extraction channel, the embeddings’ dimension is permuted and passed to stacked convolutional layers. After convolution, the output is permuted back to its original dimension, passed through another dropout layer, and then linearly projected. In parallel, global features are extracted using a Bidirectional GRU, followed by a dropout layer and a linear projection. The outputs from local and global feature extraction channels are then independently processed using an attention mechanism and subsequently normalized. The outputs of both channels are then concatenated, and a final dropout and linear projection layer are applied to the resulting features to predict entity labels.

heading: 5 Experiment and Results
text: Our model was implemented using PyTorch (Paszke et al., 2019), and Transformers from HuggingFace (Wolf et al., 2019). The model was trained for 10 epochs on an NVIDIA RTX-A6000 GPU, with a batch size of 32 and a learning rate of 1× 10−3. We used the officially provided training, validation, and test splits to train, evaluate, and test our model’s performance. We have evaluated the performance of our model based on precision, recall, and micro-F1 score, as mentioned in the shared task guideline. The experimental results are summarized in Table 2.
Algorithm 1: Attention-Based DualChannel Neural Network
1 Input: Sequence IDs, mask 2 Output: Predicted entity labels 3 he ← BERT(x); 4 hd ← Dropout(he); 5 Local Feature Extraction 6 hp ← permute(hd, 2, 1); 7 hc ← StackedConv1D(hp); 8 hp ← permute(hc, 2, 1); 9 hp ← Dropout(hp); 10 cl ← Linear(hp, dc); 11 Global Feature Extraction 12 hb ← BiGRU(hd); 13 hb ← Dropout(hb); 14 hg ← Linear(hb, dg); 15 Attention 16 a1 ← MHAttn(hg, hg, hg); 17 a1 ← LayerNorm(a1); 18 a2 ← MHAttn(cl, cl, cl); 19 a2 ← LayerNorm(a2); 20 Combine & Predict 21 f ← concat([a1, a2], dim = 2); 22 fd ← Dropout(f); 23 ŷ ← Linear(fd, C);
Our experiments showed that the base AraBERTv0.2-Twitter (Antoun et al., 2020) outperformed AraBERTv0.2. Additionally, the best-performing model was the combination of AraBERTv0.2-Twitter with Conv1D, BiGRU, and Attention, which achieved an F1 score of 89.32% on the evaluation set, highlighting the benefits of combining these techniques for this specific task.
Table 5 displays the model’s performance across different classes. High-performing entities, including PERCENT, PERS, ORDINAL, DATE, and GPE, exhibit F1 scores exceeding 93%. In contrast, entities such as PRODUCT, CURR, and UNIT have significantly lower F1 scores, underscoring the need for further improvements.
As shown in Table 3, the first place in Subtask-1 was secured by notfine_tuning, achieving a microF1 score of 91%. Our team, Addax, closely followed with a micro-F1 score of 90%, tying for second place.

heading: 6 Discussion
text: Our experimental results demonstrate that integrating a BERT-based model with CNNs, BiGRU, and an attention mechanism in a two-channel network enhances the base model’s performance. The ablation studies confirm that each component contributes to improved outcomes, validating the robustness of our architecture. Moreover, the AraBERTv0.2-Twitter model outperforms the AraBERTv0.2 model, suggesting that pretraining on Twitter data more effectively captures the structural nuances of social media text, which aligns better with the WojoodNER 2024 corpus. Future research should explore multi-head models and address low-represented classes to further enhance performance. Focusing on these areas could provide deeper insights and improvements in Arabic Named Entity Recognition (NER).

heading: 7 Conclusion
text: In this paper, we introduced the architecture we used for the NER Shared Task 2024, Subtask-1. Our model is based on a BERT language model and uses a two-channel parallel neural network with an attention mechanism. Our architecture combines a Convolutional Layers Channel and a Bidirectional GRU Channel to extract local and long-range dependencies within sentences. We found that the attention mechanism further improves the model’s performance by allowing the model to focus on relevant parts of the input sequence. As shown in our ablation studies, all components of the model contributed to enhancing the final results. Our model achieved a micro-F1 score of 90.13% on the test set for Subtask-1 of WojoodNER 2024 Shared Task, securing the second position.

heading: Future Work
text: Future research should explore multi-head models and address low-represented classes to further enhance performance. Focusing on these areas could provide deeper insights and improvements in Arabic Named Entity Recognition (NER)