SMASH at AraFinNLP2024: Benchmarking Arabic BERT Models on The Intent Detection (2024)

Abstract: The recent growth in Middle Eastern stock markets has intensified the demand for specialized financial Arabic NLP models to serve this sector. This article presents the participation of Team SMASH of The University of Edinburgh in the Multi-dialect Intent Detection task (Subtask 1) of the Arabic Financial NLP (AraFinNLP) Shared Task 2024. The dataset used in the shared task is the ArBanking77 (Jarrar et al., 2023). We tackled this task as a classification problem and utilized several BERT and BART-based models to classify the queries efficiently. Our solution is based on implementing a two-step hierarchical classification model based on MARBERTv2. We fine-tuned the model by using the original queries. Our team, SMASH, was ranked 9th with a macro F1 score of 0.7866, indicating areas for further refinement and potential enhancement of the model’s performance.

heading: 1 Introduction
text: The Financial Natural Language Processing (NLP) models have a crucial role in facilitating the daily operations offered by financial sectors worldwide. Studies show that the Middle Eastern stock markets have a robust growth in recent years (Abid et al., 2016; Jarrar, 2021), yet Arabic NLP in the financial sector still faces significant challenges (Darwish et al., 2021b; Guellil et al., 2021). Hence, advanced Arabic NLP tools become essential to address local dialect differences and meet the needs of international financial communities dealing with these markets. Despite the remarkable advancements in transformer-based English NLP models in recent years, the validation of robust financial Arabic NLP models has remained notably deficient. This disparity persists in adequately addressing the linguistic nuances inherent in the diverse Arabic dialects (Abu Farha and Magdy, 2021; Darwish et al., 2021a; Abdul-Mageed et al., 2011), thereby hindering their efficacy within the financial sector (Zman-
dar et al., 2023). This presents both challenges and opportunities for Arabic NLP researchers.
The Modern Standard Arabic (MSA) is the standard version of Arabic that is usually used in official communications, news, and books. However, there are more than 27 Arabic dialects used as spoken versions across the Arabic world, and they have emerged in a written format recently (Elgabou and Kazakov, 2017). Due to the complexity of the linguistic features that MSA and these dialects have, NLP tools fail to support them properly (Abu Farha and Magdy, 2021; Darwish et al., 2021a; AbdulMageed et al., 2011). In pursuit of enhancing financial Arabic NLP, AraFinNLP shared task 2024 (Malaysha et al., 2024) encompasses two pivotal subtasks:
• Subtask-1, Multi-dialect Intent Detection, • Subtask-2, Cross-dialect Translation and In-
tent Preservation, in the banking domain.
The dataset used in this shared task is ArBanking77 (Jarrar et al., 2023), the Arabic version of the English Banking77 dataset (Jarrar et al., 2023; Casanueva et al., 2020). ArBanking77 contains of different Arabic dialects including the MSA, Moroccan, Saudi, Palestinian and Tunisian Arabic, and contains 39,315 annotated queries within the Banking sector.
This paper describes our participation in Subtask 1 of the AraFinNLP Shared Task 2024 (Malaysha et al., 2024), where we achieved 9th place. We compared the performance of different BERT-based models in different setups, including direct finetuning on the 77 intents and a two-step classification approach in which we grouped the intents based on their categories. Our official submission was based on the two-step classification approach utilizing the MARBERTv2 model, which achieved an F1 score of 0.9470 on the Palestinian dialect validation set (PAL val) but only 0.7866 on the shared task official results on the blind test set.
403

heading: 2 Literature Review
text: Intent detection and classification is a form of text classification problem and it is widely investigated by researchers (Weld et al., 2022). Most of the works focused on English, while Arabic lagged behind with a few works such as (Bashir et al., 2018; Alruily, 2022; Jarrar et al., 2023). Bashir et al. (2018) targeted intent classification in the context of dialogue systems. Alruily (2022) developed a chatbot that utilizes a transformer model trained for named entity recognition and entity classification in a multitask learning setup. Jarrar et al. (2023) created ArBanking77 dataset that covers 77 intents in the banking domain.

heading: 3 Data
text: The dataset used in the AraFinNLP 2024 shared tasks is ArBanking77 (Jarrar et al., 2023), a translated version of the English Banking77 dataset into MSA and Palestinian Arabic (Jarrar et al., 2023; Casanueva et al., 2020). ArBanking77 dataset contains 39,315 annotated examples over 77 intents (Jarrar et al., 2023). For evaluation, the shared task organizers split the ArBanking77 dataset into multiple sets, as shown in Table 1. The training set consists of two subsets, Palestinian training "PAL train" and "MSA train," and the validation set consists of "PAL val" and "MSA val." We combined the training subsets (MSA and PAL) into one training set and the two validation sets into one validation set in our experiments.

heading: 4 Methodology
text: This section provides an overview of the experimental setup and models used in the experiments. The experiments span two approaches: direct finetuning on all 77 classes (intents) and two-step finetuning after grouping the intents into groups. In both approaches, the models were fine-tuned on the training data for the task.

heading: 4.1 Models
text: This section summarises the models we used in the experiments. Each of these models has been fine-tuned for the aforementioned task by using the
ArBanking77 dataset (Jarrar et al., 2023). Based on the literature (Abu Farha and Magdy, 2021; AbdulMageed et al., 2021; Alammary, 2022), we selected a set of models to test them in the classification. The models include the following:
AraBART (Kamal Eddine et al., 2022): an Arabic model based on BART (Lewis et al., 2020) in which the encoder and the decoder are pretrained end-to-end. It is pre-trained on a corpus of 73GB.
AraBERT: an Arabic-specific BERT model provided by (Antoun et al.). We utilized the two AraBERT models, v0.2-base and v2-base. Both models are pre-trained on 24GB of data from Wikipedia, news articles, and the Open Source International dataset (OSIAN).
mBERT (Devlin et al., 2018): a multilingual BERT model developed by Google AI and trained on 104 languages from Wikipedia’s data.
CAMeLBERT (Inoue et al., 2021): we utilized two models, the dialectal Arabic (DA) and the mixed model (Mix), which trained on mixed data of MSA, dialectal Arabic, and classical Arabic.
MARBERT (Abdul-Mageed et al., 2021): a model trained on a set of 128GB dataset, consisting of 1B tweets.
MARBERTv2 (Abdul-Mageed et al., 2021): a version of MARBERT model but trained further on 61GB of MSA data in addition to an 8.6GB of Arabic news dataset.
QARiB (Chowdhury et al., 2020; Abdelali et al., 2021): a dialectal Arabic BERT model, trained on data from tweets and a combination of Arabic GigaWord, Abulkhair Arabic Corpus, and OPUS.

heading: 4.2 Experimental Setup
text: In the experiments, we utilize the functionalities available in HuggingFace’s1 Transformers library to fine-tune the models described in 4.1. We used AutoModelForSequenceClassification, which adds a classification layer on top of a pre-trained transformer model. We fine-tuned each model for 7 epochs using the AdamW optimizer, with an initial learning rate of 2e-05. The batch size was set to 128, with the weight decay parameter of 0.01.
In the first approach, we fine-tuned the models using the PAL and MSA training datasets, and the validation datasets. We reported the performance of the models using the "PAL val" dataset.

heading: 4.3 Hierarchical Classification Experiments
text: We set up the second experiment using the hierarchical classification approach in which the classi-
1https://huggingface.co
fication decisions are made in two steps. The first step is to identify the context, which we call the group, and the second step is to identify the subclass within that group, which is mapped to the original intent in the ArBanking77 dataset. For example, we noticed that the intents: card arrival, card linking, order physical card, and card acceptance share the same context as they all are related to the bank card. Therefore, we analyzed the intents and grouped them into 11 parent classes. Table 2 shows the number of subclasses included in each parent class. The cards group has the most number of subclasses, with 23 subclasses.
Next, in each experiment, we fine-tuned each model we discussed in section 4.1, with hyperparameter settings to be 6 epochs, AdamW optimizer, and an initial learning rate of 2e-05. The batch size was set to 32, with the weight decay parameter of 0.01. We used the training datasets and validation datasets to train the first-step model that identifies the parent class (out of 11 groups). Next, for each parent class, we trained the second-step models by using the subclasses of each parent. However, we reduced the batch size to 8 in the second step of models training. By this setup, we fine-tuned 12 models from each one of the models we discuss in section 4.1, and compared their performance. Figure 1 shows the architecture of this implementation.

heading: 4.4 Dialect Translation To MSA Experiments
text: Reviewing the blind test set provided by the shared task organizers revealed that the test set contains queries in various Arabic dialects. Hence, we experimented with translating the queries from their original form to MSA before classifying them. We translated the queries by utilizing AraT5-Arabic-
Dialects-Translation2 in both the training and validation datasets. Next, we combined the translated version of the queries and the original queries in both the training and validation steps during the fine-tuning (training). However, we reported the models’ performance in this experiment using two approaches: the first evaluates the models by using the original PAL queries, and the second evaluates the models by using the translated version of the PAL queries. Furthermore, we used the same setup we followed in the hierarchical classification method we discussed in 4.3.

heading: 5 Results and Discussion

heading: 5.1 Results
text: Table 3 summarizes the performance of the models on the Palestinian validation dataset in our four experiments. Table 3 (A) illustrates the performance of each model as evaluated by the F1 score for the classification outcomes of the PAL validation set. The results show that QARiB performed best among all models with F1 0.9273, followed by MARBERT with F1 0.9270.
Furthermore, Table 3 (B) shows that the twostep method with the original datasets enhanced the performance of all models except AraBART, which has drastically dropped by 20%. Also, QARiB has been slightly affected by 0.2%, while all other models are improved. MARBERTv2 has improved by about 5%, surpassing all other models in both experiments.
Translating the queries and adding them to the
2The model is a version of AraT5 fine-tuned on MADAR corpus. Available at: https://huggingface.co/PRAli22/ arat5-arabic-dialects-translation
training and validation set during the fine-tuning improved the performance of MARBERT, QARiB, AraBERTv2, mBERT, and AraBART models. The F1-score of the latter improved significantly from 0.6248 in (A) to 0.9196, as indicated in (D). The best model when applying the translation to the fine-tuning step is MARBERT, with an F1-score of 0.9473, followed by MARBERTv2, with an F1score of 0.9320. However, the evaluation should be on the original queries of the PAL validation set. Translating the queries in the evaluation step worsens the results, as shown in Table 3 (C), which may illustrate propagated errors.
Comparing the direct fine-tuning Table 3 (A) with the two steps classification Table 3 (B, C and D) shows clearly that the two-step classification method has improved the performance as long as it has been evaluated by the original validation set (without translating it) as shown in (B) and (D).

heading: 5.2 Official Submission
text: Based on the results on the validation set, we noticed that the best is to submit the output of the best model in Table 3 (B), the two-step method with fine-tuning by the original queries. Hence, the predictions of the model MARBERTv2 were used for the official submission. Our team, SMASH, was ranked 9th with a macro F1 score of 0.7866.

heading: 5.3 Discussion
text: The results show that MARBERT and QARiB models generally perform well in the task, which is aligned with previous studies (Abu Farha and Magdy, 2021; Abuzayed and Al-Khalifa, 2021). The two-step approach has improved the performance, yet it still requires further improvement.
Translating the queries did not enhance the performance when evaluated using the translated version of the validation set. However, performance improved when the translated queries were combined with the original queries in fine-tuning the
models. This phenomenon warrants further investigation as it may derive from the enrichment of the training data with diverse contexts and a broader semantic array. Yet, this can also introduce noise and result in errors in the classification task. Additionally, semantic inaccuracies and ambiguities induced by the auto-translation process might contribute to these observations. The latter might have been avoided by using the original validation set.

heading: 6 Conclusion
text: This paper is an exploration study investigating the performance of several BERT and BART-based models for the AraFinNLP 2024 shared task. We evaluated the performance of these models in different setups, including the direct classification approach and the two-step classification approach. Our experiments exhibited that the hierarchical classification approach generally improved the performance of the models except for AraBART. However, further consideration may improve the performance, including using different models per group, which might result in an improvement in performance. Such an approach is time consuming as it requires testing all possible permutations, which we may apply in future works. Another potential improvement may be gained by experimenting with different grouping criteria.

heading: A Appendix

heading: Future Work
text: Such an approach is time consuming as it requires testing all possible permutations, which we may apply in future works. Another potential improvement may be gained by experimenting with different grouping criteria