AlcLaM: Arabic Dialectal Language Model (2024)

Abstract: Pre-trained Language Models (PLMs) are integral to many modern natural language processing (NLP) systems. Although multilingual models cover a wide range of languages, they often grapple with challenges like high inference costs and a lack of diverse nonEnglish training data. Arabic-specific PLMs are trained predominantly on modern standard Arabic, which compromises their performance on regional dialects. To tackle this, we construct an Arabic dialectal corpus comprising 3.4M sentences gathered from social media platforms. We utilize this corpus to expand the vocabulary and retrain a BERT-based model from scratch. Named AlcLaM, our model was trained using only 13 GB of text, which represents a fraction of the data used by existing models such as CAMeL, MARBERT, and ArBERT, compared to 7.8%, 10.2%, and 21.3%, respectively. Remarkably, AlcLaM demonstrates superior performance on a variety of Arabic NLP tasks despite the limited training data. AlcLaM is available at: https://github.com/amurtadha/Alclam.

text: Proceedings of The Second Arabic Natural Language Processing Conference, pages 153–159 August 16, 2024 ©2024 Association for Computational Linguistics
Pre-trained Language Models (PLMs) are integral to many modern natural language processing (NLP) systems. Although multilingual models cover a wide range of languages, they often grapple with challenges like high inference costs and a lack of diverse nonEnglish training data. Arabic-specific PLMs are trained predominantly on modern standard Arabic, which compromises their performance on regional dialects. To tackle this, we construct an Arabic dialectal corpus comprising 3.4M sentences gathered from social media platforms. We utilize this corpus to expand the vocabulary and retrain a BERT-based model from scratch. Named AlcLaM, our model was trained using only 13 GB of text, which represents a fraction of the data used by existing models such as CAMeL, MARBERT, and ArBERT, compared to 7.8%, 10.2%, and 21.3%, respectively. Remarkably, AlcLaM demonstrates superior performance on a variety of Arabic NLP tasks despite the limited training data. AlcLaM is available at: https://github.com/amurtadha/Alclam.

heading: 1 Introduction
text: Pre-trained Language Models (PLMs) utilizing selfsupervised learning techniques, such as BERT (Devlin et al., 2018a) and RoBERTa (Liu et al., 2019), have become pivotal in advancing the field of natural language processing (NLP) through transfer learning. These models have significantly enhanced performance across a variety of NLP tasks by leveraging vast amounts of textual data and extensive computational resources. However, the necessity for large corpora and the substantial computational demand, often requiring weeks of training time (Conneau et al., 2020; Raffel et al., 2020; Adiwardana et al., 2020), has primarily confined the development of such models to the English language and a few other major languages.
This limitation has sparked an increased interest in creating multilingual models capable of understanding and processing multiple languages simultaneously. Innovations such as mBERT (Devlin et al., 2018a), XLM-RoBERTa (Conneau et al., 2020) and LaBSE (Feng et al., 2022) aim to address this gap. Despite these efforts, the performance of these multilingual models typically lags behind their monolingual counterparts. This discrepancy is largely due to smaller, language-specific vocabularies and less comprehensive language-specific datasets (Virtanen et al., 2019; Antoun et al., 2020; Dadas et al., 2020; de Vries et al., 2019; Malmsten et al., 2020; Nguyen and Nguyen, 2020).
Furthermore, while languages with similar structures and vocabularies may benefit from shared representations (Conneau et al., 2020), this advantage does not extend to languages such as Arabic. Arabic’s unique morphological and syntactic structures share little in common with the morphosyntactic frameworks of more abundantly represented Latinbased languages. To address this, various Arabicspecific PLMs have been developed, including AraBERT (Antoun et al., 2020), ArBERT (AbdulMageed et al., 2021), and CAMeL (Inoue et al., 2021). These models significantly enhance Arabic NLP tasks over multilingual models. However, they are predominantly trained on Modern Standard Arabic (MSA) datasets. This focus on MSA introduces two primary limitations: first, there is reduced recognition of dialectal tokens, which vary widely across different Arabic-speaking regions; second, there is a biased weighting towards MSA tokens in the models, which may not accurately reflect the linguistic nuances present in everyday Arabic usage.
In this paper, we first introduce a new corpus of 3,372,744 Arabic dialectal texts, meticulously sourced from social media platforms such as YouTube and Facebook. Second, we outline the procedure for pretraining the transformer model
153
(Devlin et al., 2018a) specifically for the Arabic language, which we dub AlcLaM. Note that we only train AlcLaM on 13GB text due to computational resources limitation. Finally, we assess AlcLaM’s performance on three Arabic NLU downstream tasks, each distinct in nature: (i) Arabic Dialect Identification (DID), (ii) Sentiment Analysis (SA), and (iii) Hate Speech and Offensive Language Detection. Despite the limited training data, our experimental results demonstrate that AlcLaM attains state-of-the-art results on most datasets, surpassing several baseline models, including previous multilingual and single-language approaches.
In summary, our contributions are twofold.
• We constructed a massive corpus of Arabic dialects, derived from the content and comments on Arabic pages on Facebook and videos from Arabic-speaking YouTubers. This corpus represents a rich variety of regional dialects and everyday language usage that has been underrepresented in previous models.
• We developed an Arabic pre-trained language model, namely AlcLaM, specifically optimized to handle the diversity and complexity of Arabic dialects based on the newly created corpus, enhancing its applicability across a wider range of NLP tasks involving Arabic text.

heading: 2 Related Work
text: Pre-trained language models (PLMs) using a selfsupervised masking objective, such as BERT (Devlin et al., 2018a) and RoBERTa (Liu et al., 2019), have significantly advanced NLP. These models have multilingual versions, including mBERT (Devlin et al., 2018a), XLM-RoBERTa (Conneau et al., 2020) and LaBSE (Feng et al., 2022). Additionally, models featuring different objectives or architectures, such as ALBERT (Lan et al., 2020), T5 (Raffel et al., 2020), its multilingual variant mT5 (Xue et al., 2021), and GPT-3 (Brown et al., 2020), LLaMA (Touvron et al., 2023), PaLM (Chowdhery et al., 2023), GPT-4 (OpenAI, 2023), and RoFormer (Su et al., 2024) have been introduced.
Non-English PLMs have also been developed. These include Bertje for Dutch (de Vries et al., 2019), CamemBERT (Martin et al., 2020) and FlauBERT (Le et al., 2020) for French, PhoBERT for Vietnamese (Nguyen and Nguyen, 2020), as well as models for Finnish by Virtanen et al. (2019),
for Polish by Dadas et al. (2020), and for Swedish by Malmsten et al. (2020). Pyysalo et al. (2021) have created monolingual LMs using Wikipedia data for 42 languages. For Arabic, MSA-based PLMs includes AraBERT (Antoun et al., 2020) ArabicBERT (Safaya et al., 2020), ArBERT (AbdulMageed et al., 2021). Another line of research involves pre-training models on a combination of MSA and dialectal data, such as MDBERT (AbdulMageed et al., 2021) and CAMeL (Mubarak et al., 2021). Our contributions to this field include a comprehensive Arabic dialectal corpus spanning various dialects and the development of an Arabic PLM. Our model, named AlcLaM, enhances the representation of linguistic diversity in Arabic NLP.

heading: 3 Methodology
text: In this paper, we develop AlcLaM, an Arabic dialect language representation model that enhances the performance on several Arabic NLP tasks. This model builds upon the BERT architecture, a stacked Bidirectional Transformer Encoder (Devlin et al., 2018a). Recognized as the foundation for many state-of-the-art results in various NLP tasks across multiple languages, BERT’s architecture has proven highly effective. Below, we detail the dialectal corpus used for AlcLaM’s pretraining, the pretraining setup, and the fine-tuning process.

heading: 3.1 Arabic Dialectal Corpus.
text: The original BERT model was trained on a corpus comprising 3.3 billion words extracted from English Wikipedia and the Book Corpus (Zhu et al., 2015). Due to the comparatively smaller size of Arabic Wikipedia dumps in comparison to English ones, we opted to utilize Arabic text from the English-Arabic bilingual corpora of opensubtitles 1 (Itamar and Itai, 2008).
It is noteworthy that publicly available Arabic corpora are heavily dominated by MSA, while social media and online reviews predominantly feature Arabic dialects. This creates a bias towards MSA tokens in Arabic PLMs, potentially leading to tokenizers failing to recognize a significant portion of dialectal vocabulary. To address this, we manually scraped Arabic texts from social media platforms. Initially, we scrape posts and comments from popular Arabic YouTube channels and Facebook Pages. However, we observed that many of
1opensubtitles
these comments consisted of verses from the Holy Quran and Hadith, typically written in MSA. Since our focus was on dialectal texts, we trained a binary classifier (MSA-Dialect) to filter out MSA texts. Specifically, we treated all dialectal instances of the MADAR corpus as one class, labeled "Dialect" (Bouamor et al., 2019; Murtadha et al., 2022), and utilized it to fine-tune the CAMeL model (Inoue et al., 2021), which achieved a remarkable 98% accuracy. Our final corpus comprises 3,372,744 dialectal sentences with 54,557,408 tokens. To the best of our knowledge, this marks the first attempt to assemble such a comprehensive Arabic dialectal corpus.

heading: 3.2 Model Training
text: For AlcLaM, we adhere to the original BERT (Devlin et al., 2018a). Each training input sequence is generated using whole word masking, where 15% of the N input tokens are chosen for replacement. These selected tokens undergo replacement as follows: 80% are substituted with the [MASK] token, 10% with a random token, and 10% remain unchanged. Following Liu et al. (2019), we exclude the next sentence prediction (NSP) loss from our training process. This decision is based on the observation that removing the NSP loss either matches or slightly improves downstream task performance. We employ the same network configuration as BERT-base: consisting of 12 layers, 768 hidden units, and 12 attention heads, resulting in approximately 125 million parameters. During training, we utilize a batch size of 64 sequences and set a maximum sequence length of 128 tokens and 5 training epochs. Throughout training, we set the learning rate to 5e´ 5.

heading: 3.3 Fine-tuning
text: To fine-tune AlcLaM for sequence classification, we utilize the final hidden state of the first token, corresponding to the embedding of the special “[CLS]” token that is prepended to the beginning of each sentence (Murtadha et al., 2024). A simple feed-forward layer with a Softmax activation function is added to compute the probability distribution over the predicted output classes. During fine-tuning, both the classifier and the pre-trained model weights are jointly trained to maximize the log-probability of the correct class (Ahmed et al., 2023).

heading: 4 Empirical Evaluation

heading: 4.1 Datasets
text: We evaluated AlcLaM on the following datasets that cover various NLP tasks in Arabic. Sentiment analysis (SemEval 2017 task 4 (Kiritchenko et al., 2016), ASAD (Alharbi et al., 2020), ASTD (Nabil et al., 2015), ArSAS (Elmadany et al., 2018), LABR (Aly and Atiya, 2013)), offensive language detection (Adult (Mubarak et al., 2021), Offensive and HateSpeech (Mubarak et al., 2020)), dialect identification (MADAR-6, MADAR-26(Bouamor et al., 2019) and NADI (Abdul-Mageed et al., 2020)). For experiments, MADAR-2 and MADAR9 are derived from MADAR-26. MADAR-2 is binary (MSA-dialect), while MADAR-9 categorizes dialects into 9 regions: Yemen, MSA, Maghreb, Nile Egypt, Libya, Gulf, Nile Sudan, Iraq, and Levant.

heading: 4.2 Baselines
text: We compare our AlcLaM model with:
1. Multilingual PLMs like mBERT (Devlin et al., 2018b) and LaBSE (Feng et al., 2022);
2. MSA-based Arabic PLMs such as AraBERT (Antoun et al., 2020) and ArBERT (AbdulMageed et al., 2021);
3. MSA-Dialect-based PLMs, including MdBERT (Talafha et al., 2020), and MARBERT (Abdul-Mageed et al., 2021) and CAMeL (Inoue et al., 2021).

heading: 4.3 Results
text: For each dataset, we report the average results of five runs, each with different random seeds, to ensure statistical significance. The results for various Arabic NLP tasks are presented in Table 1 and Table 2 in terms of F1 and accuracy metrics, respectively. From these results, we make the following observations:
1. Multilingual models such as mBERT and LaBSE are outperformed by Arabic-specific models that are pre-trained with larger vocabularies and more extensive language-specific datasets. This observation aligns with the findings of Abdul-Mageed et al. (2021).
2. Models that incorporate dialectal data during pre-training, such as MdBERT, CAMeL and MARBERT, not only excel in DID task, but
also perform significantly across a broader range of Arabic NLP tasks. This suggests that the similarities among Arabic dialects may not always have positive effects on other tasks beyond ADI. The experimental results underscore the value of integrating more dialectal information during training, as the tokenizers in these models are likely to recognize more dialect-specific tokens, which are often unidentified in other models.
3. Despite being trained on less MSA text and fewer training steps, due to computational resource constraints, our model outperforms its alternatives in most tasks and achieves competitive performance in others. Although the improvements in tasks other than ADI are modest, they are significant given the inherent complexities of the Arabic language.
In tasks beyond DID, AlcLaM may show mod-
est improvements, but it introduces vital empirical factors like stability and statistical significance, supported by a t-test (p ă 0.05). MSA-Dialect PLMs consistently demonstrate superior performance across a range of Arabic NLP tasks. These empirical findings clearly support our claim regarding the critical importance of incorporating Arabic dialectal data in the pre-training process.

heading: 5 Conclusion
text: In this paper, we present AlcLaM, a novel BERTbased model trained specifically to address the challenge of Arabic dialectal variation. Leveraging a carefully curated corpus sourced from social media platforms, AlcLaM its alternatives across various Arabic NLP tasks, despite being trained on significantly less data. For future work, expanding the dialectal vocabulary without increasing inference costs, inspired by Chinese character modeling.
Limitations
Despite the advancements achieved by AlcLaM, it is important to acknowledge its current limitations:
• AlcLaM is trained from scratch to build its vocabulary. However, incorporating weights of new dialectal vocabulary from existing Arabic PLMs and adjusting through continued training is a potential avenue for enhancement. Nevertheless, expanding the vocabulary size to encompass more dialectal tokens might lead to increased inference costs.
• Given that AlcLaM was trained on approximately 10% of the training data used by its alternatives, due to computational resource constraints, its performance on generative tasks may not be as significant. Nonetheless, this limitation can be mitigated by continued training on our open-source AlcLaM model.

heading: Future Work
text: For future work, expanding the dialectal vocabulary without increasing inference costs, inspired by Chinese character modeling. Limitations
Despite the advancements achieved by AlcLaM, it is important to acknowledge its current limitations:
• AlcLaM is trained from scratch to build its vocabulary. However, incorporating weights of new dialectal vocabulary from existing Arabic PLMs and adjusting through continued training is a potential avenue for enhancement. Nevertheless, expanding the vocabulary size to encompass more dialectal tokens might lead to increased inference costs. • Given that AlcLaM was trained on approximately 10% of the training data used by its alternatives, due to computational resource constraints, its performance on generative tasks may not be as significant. Nonetheless, this limitation can be mitigated by continued training on our open-source AlcLaM model