SemanticCuetSync at AraFinNLP2024: Classification of Cross-Dialect Intent in the Banking Domain using Transformers (2024)

Abstract: Intention detection is a crucial aspect of natural language understanding (NLU), focusing on identifying the primary objective underlying user input. In this work, we present a transformer-based method that excels in determining the intent of Arabic text within the banking domain. We explored several machine learning (ML), deep learning (DL), and transformer-based models on an Arabic banking dataset for intent detection. Our findings underscore the challenges that traditional ML and DL models face in understanding the nuances of various Arabic dialects, leading to subpar performance in intent detection. However, the transformer-based methods, designed to tackle such complexities, significantly outperformed the other models in classifying intent across different Arabic dialects. Notably, the AraBERTv2 model achieved the highest micro F1 score of 82.08% in ArBanking77 dataset, a testament to its effectiveness in this context. This achievement, which contributed to our work being ranked 5 in the shared task, AraFinNLP2024, highlights the importance of developing models that can effectively handle the intricacies of Arabic language processing and intent detection.

heading: 1 Introduction
text: Textual intent detection is a natural language processing (NLP) process that involves identifying and understanding a text input’s underlying purpose or objective. This task is essential for enabling machines to comprehend what a user wants to achieve with their message. The goal is to classify the text into predefined intent categories, such as asking a question, requesting, providing feedback, or initiating a transaction. Intent detection plays a pivotal role in enhancing user experiences, automating processes, ensuring accuracy, scaling operations, providing business insights, and supporting multilingual environments. It is a cornerstone technology
for creating intelligent and responsive systems in various domains. In the banking sector, intent detection is vital for improving customer service, enhancing operational efficiency, ensuring accuracy and security, gaining data-driven insights, supporting a multilingual customer base, and maintaining regulatory compliance. It helps banks deliver superior service while managing risks and optimizing operations.
This research approach considered intent detection a classification problem, aiming to assign a predetermined intent label to the user’s input. While intent detection in English has been extensively studied (Kumar et al.; Wang et al., 2024; Zhang et al., 2021), the task is challenging for low-resource languages such as Arabic and its dialects. Very little research has been conducted on intent detection in Arabic banking, making creating precise and efficient systems challenging. Especially in particular fields like banking, the lack of research has led to fewer datasets, tools, and models customized in Arabic. As a result, this gap restricts the development of efficient NLU systems in the financial sector for Arabic-speaking users. The AraFinNLP shared task aims to bridge this gap. Under this shared task, we developed a transformer-based system to detect intent in various Arabic dialects within the banking domain. The system was evaluated on the ArBanking77 dataset, which consists of 77 classes that cover different Arabic dialects, including Palestinian Arabic and Modern Standard Arabic (MSA). The main contributions of this work are:
• Proposed a fine-tuned transformer model for intent detection in numerous Arabic dialects.
• Investigated the performance of nine classification models (Random Forest (RF), Multinomial Naive Bayes (MNB), Support Vector Machine (SVM), CNN, CNN+LSTM, CNN+BiLSTM, AraBERTv2,
422
BERT-Banking77, CamelBert) to discover a reasonable model for intent detection in Arabic.

heading: 2 Related Work
text: Many studies have been accomplished on textual intent detection in recent years. Yolchuyeva et al. (2020) proposed self-attention networks, including SAN and BiLSTM for intent detection. They used 300-dimensional Word2Vec and FastText embeddings pre-trained on English Wikipedia as word representations. Their model achieved an F1 score of 96.81. Zhang et al. (2020b) presented transformer-based solutions for English and multilingual datasets. This model also demonstrated vital zero and few-shot performance, reaching over 75% accuracy using only 100 training examples in all datasets. A dual channel model for intent detection was proposed by Wang et al. (2022). It creates a dual-channel feature extraction technique by combining the high-level features obtained from the pooling operation and the capsule network. The final intent detection is achieved by utilizing the feature fusion of the two channels. Their model achieved an F1 score of 88.5% and 89.76% in ATIS and SMP2019-ECDT datasets, respectively. Mezzi et al. (2022) suggested a system that uses the concepts of intent recognition to make mental health diagnoses of Arabic-speaking patients based on bidirectional encoder representations from the transformers (BERT) model. Their system achieved an F1 score of 94%. Shams et al. (2022) fine-tuned several BERT models to detect intent for Urdu. Their analysis shows that on their two datasets, the fine-tuned models of mBERT and RoBERTa-urdusmall reach 96.38% and 93.30% accuracy, respectively. The intent detection problem can be treated as a classification problem as well. Aldjanabi et al. (2021) proposed a multi-task learning technique developed on a pre-trained Arabic language model for determining offensive and hate speech. The developed model surpassed past methods for offensive and hate speech detection tasks. Jarrar et al. (2023a) proposed a solution for detecting intent in modern and dialectical Arabic using a neural model based on AraBERT. Their proposed model achieved F1 scores of 0.9209 and 0.8995 on MSA and Palestinian dialect, respectively. Considering the current constraints of the task, this work applies a transformer-based method to identify intent in various Arabic dialects used in the banking sec-
tor.

heading: 3 Dataset and Task Description
text: This work used ArBanking77 (Jarrar et al., 2023b), which originated from the Banking77 dataset (Casanueva et al., 2020). It includes 77 different classes (intents) and features queries in MSA and Palestinian (PAL) dialect. Table 1 shows the distribution of train, development, and test sets for MSA and PAL. The dataset comprises 21,559 texts in the training set, 2,464 in the development set, and 11,721 in the test set.
Subtask-1 (Malaysha et al., 2024) concerns cross-dialect intent detection in the banking domain. This subtask mainly focused on creating systems that accurately classify consumer intents from queries in multiple Arabic dialects. Table 2 illustrates a few examples of the query and intent of the dataset.

heading: 4 System Overview
text: This work exploited a variety of ML, DL, and transformer-based models to conduct multi-dialect intent detection in the Arabic banking industry. Figure 1 illustrates the schematic configuration of the proposed technique.
Textual data was converted into highdimensional vectors using BoW and TF-IDF (Term Frequency-Inverse Document Frequency)
(Takenobu, 1994) in ML-based techniques. For DL models, Word2Vec and FastText word embeddings were utilized (Mikolov et al., 2013), while BERTbased tokenizers were employed for transformer models.

heading: 4.1 Classifiers
text: This work explored nine classification models (3 ML, 3 DL, and 3 transformers).
• ML Models: This work tested RF (Breiman, 2001), SVM (Boser et al., 1992), and MNB with TF-IDF and BoW features. Table 3 shows several ML model implementation parameters.
• CNN: The CNN model used in this work includes an embedding layer with an output dimension of 256. It comprises one convolution layer with 512 filters and a global maximum pooling layer for downsampling. The dense layer with L2 regularization introduces nonlinearity to prevent overfitting. The output layer allows for multi-class classification with its 77 units and softmax activation. The loss function used here was ‘SparseCategoricalCrossentropy’ with the ‘Adam’ optimizer.
• CNN+LSTM: This approach used one convolution layer, one maximum pooling layer,
three LSTM layers, and three dense layers with a dropout layer. The number of units for the convolution layer was 512; for the LSTM layers, they were 512, 256, and 128, respectively. For the first dense layer, the unit size was 256 with ‘relu’ activation; the second dense layer had 128 units with ‘tanh’ activation, and the third dense layer was for classification with 77 units.
• CNN+BiLSTM: This model employs a comparable structure to the CNN+LSTM model but it incorporates Bidirectional LSTM.
• AraBERTv2: AraBERTv2 (Antoun et al., 2020) is the successor of AraBERT, which leverages the BERT architecture. This model is evaluated on different downstream tasks like sentiment analysis, named entity recognition, and Arabic question answering. This work fine-tuned AraBERTv2 on the Arabic banking dataset and obtained a decent outcome.
• BERT-Banking77: BERT-Banking771 detects intent in English and is fine-tuned using BANKING77 (Casanueva et al., 2020) dataset. In this approach, we used ‘HelsinkiNLP/opus-mt-ar-en’ (Zhang et al., 2020a) and ‘llama-3-8b-instruct’ (Touvron et al., 2023) for zero-shot translation before feeding the input text into the model.
• Arabic CamelBERT: CamelBERT (Inoue et al., 2021) was built by fine-tuning BERT on different dialects of the Arabic language for domain-specific tasks. It used the ASTD, ArSAS, and SemEval datasets for fine-tuning. This model understands the Arabic dialects well and was further fine-tuned in this work on the Arabic banking dataset.
Table 4 illustrates different hyperparameters such as learning rate (LR), weight decay (WD), warmup steps (WS), and number of epochs (EP) used in the transformer-based models.
1https://huggingface.co/philschmid/BERT-Banking77

heading: 5 Results and Analysis
text: Table 5 demonstrates the evaluation results of ML, DL, and transformer-based models on the test set. The superiority of the models is determined based on the F1-score).
Results revealed that SVM with TF-IDF features earned the most elevated F1-score (61.69%) among the ML approaches. On the other hand, CNN outperformed the CNN+LSTM (38.50%) and CNN+BiLSTM (42.04%) by obtaining the highest F1-score of 43.33%, which is approximately 18.36% lower than the best ML approach
(SVM). DL models utilizing FastText word embeddings also performed relatively poorly, with CNN, CNN+LSTM, and CNN+BiLSTM achieving F1-scores of 40.52%, 39.72%, and 37.82%, respectively.
However, the transformer model outperforms the top-scoring ML and DL models. AraBERTv2 achieved the highest F1 score of 82.08%, whereas Bert-Banking77 and CamelBert scored 68.56% and 73.00%, respectively.
While ML and DL models performed well on the development set, their performance declined on the test sets. DL models significantly underperformed compared to ML models. AraBERTv2 is pre-trained with Wikipedia dataset, encompassing a range of Arabic dialects and demonstrating superior performance over all exploited models.

heading: 5.1 Error Analysis
text: A comprehensive quantitative and qualitative error analysis was conducted to provide detailed insights into the proposed model’s performance.
Quantitative Analysis
Figure 2 categorizes intents into correctly classified and misclassified. Out of 11,721 intents, 9,621 (82%) were accurately predicted, while 2,100 (18%) were misclassified. Although the test dataset includes various Arabic dialects, the proposed model was trained only on the PAL and MSA datasets. This limitation might contribute to the model’s difficulty in generalizing to unseen Arabic dialects, leading to several misclassifications.
Qualitative Analysis Table 6 presents some predicted outputs of the proposed model. In the first and third samples, the model successfully predicted the intent of the text. On the other hand, it failed to do so in the second sample. This is challenging to predict because queries regarding top up limits and top up reverted can be very similar and difficult to differentiate.

heading: 6 Conclusion
text: This study investigated the efficacy of several transformer-based, DL, and ML models on the Arabic banking dataset to detect intent. The results demonstrate that the transformer-based model (AraBERTv2) performed superiorly in subtask-1, achieving the highest F1 score of 82.08%. The study suggests that one of the main reasons for the proposed model’s poor performance is its inability to generalize to unseen Arabic dialects. Future research could investigate advanced large language models (LLMs) to achieve better performance.

heading: Future Work
text: Future research could investigate advanced large language models (LLMs) to achieve better performance