ORCA: A Challenging Benchmark for Arabic Language Understanding
Year: 2023

Abstract:
Due to the crucial role pretrained language models play in modern NLP, several benchmarks have been proposed to evaluate their performance. In spite of these efforts, no public benchmark of diverse nature currently exists for evaluating Arabic NLU. This makes it challenging to measure progress for both Arabic and multilingual language models. This challenge is compounded by the fact that any benchmark targeting Arabic needs to take into account the fact that Arabic is not a single language but rather a collection of languages and language varieties. In this work, we introduce a publicly available benchmark for Arabic language understanding evaluation dubbed ORCA. It is carefully constructed to cover diverse Arabic varieties and a wide range of challenging Arabic understanding tasks exploiting 60 different datasets (across seven NLU task clusters). To measure current progress in Arabic NLU, we use ORCA to offer a comprehensive comparison between 18 multilingual and Arabic language models. We also provide a public leaderboard with a unified single-number evaluation metric (ORCA score) to facilitate future research.1

heading: 1 Introduction
text: Pretrained language models (PLMs) (Devlin et al., 2019; Liu et al., 2019b; Lan et al., 2019; Zhang et al., 2019; Sanh et al., 2019; Radford et al., 2019; Dai et al., 2019; Clark et al., 2020; Lewis et al., 2020a; Zaheer et al., 2020; Brown et al., 2020; Beltagy et al., 2020; Zhang et al., 2020a; Conneau et al., 2020a; Kitaev et al., 2020; Zhang et al., 2020b; He et al., 2021; Le et al., 2021; Raffel et al., 2022; Chung et al., 2022; Chowdhery et al., 2022) have become a core component of the natural language understanding (NLU)
1https://orca.dlnlp.ai/. ⋆All authors contributed equally.
pipeline, making it all the more important to evaluate their performance under standardized conditions. For this reason, several English-based benchmarks such as GLUE (Wang et al., 2018), SuperGLUE (Wang et al., 2019), SyntaxGym (Gauthier et al., 2020), Evaluation Harness (Gao et al., 2021), GEM (Gehrmann et al., 2021), NLAugmenter (Dhole et al., 2021), Dynabench (Kiela et al., 2021), MMLU (Hendrycks et al., 2021), NATURAL INSTRUCTIONS (Mishra et al., 2022), BIG-bench (Srivastava et al., 2022), and multilingual benchmarks such as XTREME (Hu et al., 2020), XGLUE (Liang et al., 2020), and MASAKHANE (Nekoto et al., 2020) have been introduced. Benchmarks for a few other languages also followed, including FLUE (Le et al., 2020) for French, CLUE (Xu et al., 2020) for Chinese, IndoNLU (Wilie et al., 2020) for Indonesian, KorNLI and KorSTS (Ham et al., 2020) for Korean, and JGLUE(Kurihara et al., 2022). This leaves behind the majority of the world’s languages, and relies on multilingual benchmarks which often have limited coverage of dialects and naturally-occurring (rather than machine translated) text. This motivates us to introduce a benchmark for Arabic. One
9559
other reason that lends importance to our work is that Arabic is a rich collection of languages with both standard and dialectal varieties and more than 400M native speaker population.
To the best of our knowledge, there have only been two attempts to provide Arabic NLU evaluation benchmarks. These are ARLUE (AbdulMageed et al., 2021) and ALUE (Seelawi et al., 2021). Although useful, both of these have major limitations: ALUE has modest coverage (only eight datasets covering only three task clusters) and ARLUE involves datasets that are not publicly available. Our goal is to rectify these limitations by introducing ORCA, which expands task coverage using fully public datasets, while also offering an accessible benchmark with a public leaderboard and processing tools as well as wide geographical and linguistic coverage. ORCA exploits 60 different datasets, making it by far the most extensive benchmark for Arabic NLU and among the most extensive for any language. We present detailed analyses of the data comprising ORCA and evaluate a wide range of available pretrained language models (PLMs) on it, thus offering strong baselines for future comparisons.
In summary, we offer the following contributions: (1) We introduce ORCA, an extensive and diverse benchmark for Arabic NLU. ORCA is a collection of 60 datasets arranged into seven task clusters, namely: sentence classification, text classification, structured prediction, semantic similarity, natural language inference (NLI), question-answering (QA), and word sense disambiguation (WSD). (2) We provide a comprehensive comparison of the performance of publicly available Arabic PLMs on ORCA using a unified ORCA score. (3) To facilitate future work, we design a public leaderboard for scoring PLMs on ORCA. Our leaderboard is interactive and offers rich meta-data about the various datasets involved as well as the language models we evaluate. (4) We distribute ORCA with a new modular toolkit for pretraining and transfer learning for NLU. The toolkit is built around standard tools including PyTorch (Paszke et al., 2019) and HuggingFace datasets hub (Lhoest et al., 2021).
The rest of the paper is organized as follows: In Section 2, we provide an overview of related work. Section 3 introduces ORCA, our Arabic NLU benchmark. In Section 4, we describe multilingual and Arabic pretrained language models we evaluate on ORCA, providing results of our
evaluation in Section 5. Section 6 is an analysis of model computational cost as measured on ORCA. We conclude in Section 7.

heading: 2 Related Work
text: Most recent benchmarks propose a representative set of standard NLU tasks for evaluation. These can be categorized into English-centric, multilingual, Arabic-specific, and X-specific (X being a language other than English or Arabic such as Chinese, Korean, or French). We briefly describe each of these categories next. We also provide a comparison of benchmarks in the literature in terms of task clusters covered and number of datasets in Table 1.

heading: 2.1 English-Centric Benchmarks
text: GLUE. The General Language Understanding Evaluation (GLUE) benchmark (Wang et al., 2018) is one of the early English benchmarks. It is a collection of nine publicly available datasets from different genres. GLUE is arranged into three task clusters: sentence classification, similarity and paraphrase, and NLI. SuperGLUE. Wang et al. (2019) propose SuperGLUE, a benchmark styled after GLUE with a new set of more challenging tasks. SuperGLUE is built around eight tasks and arranged into four task clusters: QA, NLI, WSD, and coreference resolution. The benchmark is accompanied by a leaderboard with a single-number performance metric (i.e., the SuperGLUE score).

heading: 2.2 Multilingual Benchmarks
text: bAbI. Early attempts to create multilingual benchmarks are limited in their language coverage. An example is bAbI (Weston et al., 2015), which covers only English and Hindi. It consists of a set of 20 tasks for testing text reasoning and understanding using different question-answering and coreference resolution strategies. XGLUE. XGLUE is a cross-lingual benchmark proposed by Liang et al. (2020) to evaluate the performance of PLMs. It provides 11 tasks in both NLU and NLG scenarios that cover 19 languages. The XGLUE tasks are arranged into four understanding tasks (structured predictions, text classifications, QA, NLI, semantic search), and two generation tasks (question and title generation). XTREME. The Cross-lingual TRansfer Evaluation of Multilingual Encoders (XTREME) (Hu et al., 2020) is a benchmark for evaluating the
2
2020). ALUE: (Seelawi et al., 2021). ARLUE: (Abdul-Mageed et al., 2021). ORCA: Our proposed Arabic NLU benchmark.
cross-lingual generalization capabilities of multilingual models. It covers 40 languages and includes nine datasets across four task clusters: classification (i.e., NLI and paraphrase), structured prediction (i.e., POS and NER), question answering, and sentence retrieval. Ruder et al. (2021) extend XTREME to XTREME-R (for XTREME Revisited). This new benchmark has an improved set of ten NLU tasks (including language-agnostic retrieval tasks). XTREME-R covers 50 languages. Authors also provide a multilingual diagnostic suite and evaluation capabilities through an interactive public leaderboard. Big-bench. The Beyond the Imitation Game Benchmark or shortly BIG-bench (Srivastava et al., 2022) is a collaborative2 NLP benchmark aimed to explore and evaluate the capabilities of large language models. It currently consists of 204 advanced NLP tasks, from diverse topics such as common-sense reasoning, linguistics, childhood development, math, biology, physics, social bias, and software development.3

heading: 2.3 Arabic-Specific Benchmarks
text: ALUE. To the best of our knowledge, two benchmarks for Arabic currently exist, ALUE (Seelawi et al., 2021) and ARLUE (Abdul-Mageed et al., 2021). ALUE is focused on NLU and comes with
2Contributed by 444 authors across 132 institutions. 3We exclude the Big-Bench benchmark from Table 1 because it has a very large number of tasks that we cannot fit into the table. It also involves task clusters that are unrelated to the ORCA benchmark.
eight datasets arranged into three task clusters: sentence classification, NLI, and similarity and paraphrase. The sentence classification cluster involves five datasets for offensive and hate speech detection, irony prediction, sentiment analysis, and dialect identification. The NLI cluster involves two datasets, both for NLI aiming at predicting whether a premise sentence contradicts, entails, or is neutral toward a hypothesis sentence. ALUE has one dataset for semantic similarity comprising a collection of questions pair labelled with "1" (semantically similar) or "0" otherwise. The task is to predict these similarity scores. While datasets in ALUE are publicly available and the benchmark is accompanied by a leaderboard, its size and diversity (geographical and linguistic) are modest. ARLUE. (Abdul-Mageed et al., 2021) also targets Arabic NLU tasks and is composed of 42 datasets arranged into four task clusters: sentence classification, text classification, structured prediction, and QA. Many of the datasets in ARLUE, however, are not publicly available which presents a barrier to widespread adoption. Nor is ARLUE accompanied by a leaderboard. ORCA ameliorates these challenges.

heading: 2.4 X-Specific Benchmarks
text: Other X-specific benchmarks include CLUE. (Xu et al., 2020), FLUE. (Le et al., 2020), IndoNLU. (Wilie et al., 2020), JGLUE. (Kurihara et al., 2022), and KorNLI and KorSTS. (Ham et al., 2020). We review these benchmarks in Ap-
3
pendix B.

heading: 3 ORCA Benchmark
text: We present ORCA, a benchmark for Arabic NLU that is challenging and diverse. ORCA involves 60 datasets arranged into 29 tasks and seven task clusters. In the following, we will first introduce our design principles for developing ORCA then introduce the different task clusters covered.

heading: 3.1 Design Principles
text: Our goal is to offer a challenging and diverse NLU benchmark that allows evaluation of language models and measurement of progress on Arabic NLU. To this end, we develop ORCA with a number of design principles in mind. We explain these here. Large number of public tasks. We strive to include as many tasks as possible so long as data for these tasks are public. This makes it possible for researchers to train and evaluate on these tasks without having to pay for private data. ORCA involves 60 different datasets that are all publicly available. Challenging benchmark. We design ORCA to require knowledge at various linguistic levels, making it challenging. This includes knowledge at the level of tokens in context as well as at the levels of complete sentences, inter-sentence relations, whole paragraphs, and entire documents.
Coherent task clusters and tasks. Rather than listing each group of datasets representing a given task together, we group the various tasks into task clusters. This makes it easy for us to present the various downstream tasks. It also makes it possible to derive meaningful insights during evaluation. For example, one can compare performance at a lower-level task cluster such as structured prediction to that of performance at a higher-level cluster such as natural language inference. Within the clusters themselves, we also maintain coherent subgroupings. For example, since sentiment analysis has been one of the most popular tasks in Arabic NLP, we assign it its own sub-cluster within the sentence classification cluster. Similarly, we keep tasks such as hate speech and emotion detection that exploit social media data into a single social meaning cluster. Wide linguistic variability and geographical coverage. We strive to include tasks in various Arabic varieties. This involves Modern Standard Arabic (MSA) and dialectal Arabic (DA). We include datasets collected from wide regions within the Arab world. This not only pertains our DA datasets, many of which come from various Arab countries, but also our MSA datasets as these are extracted from several news outlets from across the Arab world. This also ensures variability in topics within these datasets. To illustrate linguistic diversity within ORCA, we run an in-house binary MSA-dialect classifier on all ORCA data splits (i.e., Train, Dev, and Test).4 For a deeper understanding of ORCA data, we also calculate several measures including the average, median, mode, and typetoken ratio (TTR) of the sentences in each task. Table 3 shows the MSA vs. DA data distribution and the statistical description of ORCA datasets.
In addition, we acquire a country-level dialect distribution analysis over the data using AraT5 model (Nagoudi et al., 2022) fine-tuned on the ORCA dialect country-level dataset (DIA-C). We run this country-level classifier only on the dialectal portion of ORCA (i.e., datasets of tweets predicted as dialectal with our in-house MSA-dialect classifier). Figure F.1 shows that ORCA datasets are truly diverse from a geographical perspective.5
Accessible evaluation. To facilitate evaluation in
4As our classifier is trained using ORCADA, we exclude the ORCA dialect component from this analysis.
5Again, the country-level classifier is also trained using ORCADIA, so we exclude the dialect tasks from this analysis.
4
a reasonable time frame in a GPU-friendly setting, we cap data sizes across our Train, Dev, and Test splits to 50k, 5k, 5k samples respectively. This allows us to avoid larger data sizes in tasks such as Age and Gender that have 1.3m, 160k, 160k samples for the Train, Dev, and Test splits each and both the sentiment and dialect country-level tasks that have 190k, 6.5k, 44.2k and 711.9k, 31.5k, 52.1k for the Train, Dev, and Test data (respectively). Table 2, shows a summary of the data splits across tasks and task clusters in ORCA. Simple evaluation metric. We adopt a simple evaluation approach in the form of an ORCA score. The ORCA score is simply a macro-average of the different scores across all tasks and task clusters, where each task is weighted equally. Modularity. We design ORCA to allow users to score models on the whole benchmark but also on individual task clusters. In both cases, the leaderboard returns results averaged across the datasets within either the whole benchmark or the individual tasks (sub-leaderboards). This allows us to invite submissions of dedicated models that take as its target subsets of ORCA datasets. Figure 2 shows ORCA’s main screen with models sorted by ORCA score. We provide more screenshots illustrating ORCA’s modularity in Appendix D. Public leaderboard. We allow scoring models against ORCA through an intuitive, easy-to-use leaderboard. To facilitate adoption, we also provide a Google Colab notebook with instructions for finetunining any model on ORCA tasks. Faithful evaluation. For each submission, we require users to provide meta-data such as the number of parameters, amount of pretraining data, and
number of finetuning epochs. This facilitates comparisons across the different models. We make this meta-data available via our interactive leaderboard. Proper credit for individual dataset authors. One issue with evaluation benchmarks is that once a benchmark is created there is a concern of not giving credit to original datasets. To overcome this limitation, we distribute a simple text file with bibliographic entries for all papers describing the 60 datasets in ORCA and strongly encourage all future use to cite them.

heading: 3.2 Tasks and Task Clusters
text: As explained, we arrange ORCA into 7 task clusters. These are (1) sentence classification, (2) structured prediction (3) semantic textual similarity and paraphrase, (4) text classification, (5) natural language inference, (6) word sense disambiguation, and (7) question answering. Sentence Classification. This cluster involves the following sentence-level classification tasks: (1) Sentiment Analysis: 19 publicly available sentiment datasets have been used to construct this task. We merge 17 datasets benchmarked by AbdulMageed et al. (2021) with two new datasets: Arabizi sentiment analysis dataset (Fourati et al., 2020) and AraCust (Almuqren and Cristea, 2021), a Saudi Telecom tweets corpus for sentiment analysis. (2) Social Meaning: Refers to eight social meaning datasets covering prediction of hate speech and offensive language (Mubarak et al., 2020), dangerous speech (Alshehri et al., 2020), sarcasm (Farha and Magdy, 2020), adult content Mubarak et al. (2021), irony (Ghanem et al., 2019), emotion, age and gender (Mohammad et al.,
5
2018; Abdul-Mageed et al., 2020b). (3) Dialect Identification: Six datasets are used for dialect classification. These are ArSarcasmDia (Farha and Magdy, 2020), the Arabic Online Commentary (AOC) dataset (Zaidan and Callison-Burch, 2014), NADI-2020 (Abdul-Mageed et al., 2020a), MADAR (Bouamor et al., 2019), QADI (Abdelali et al., 2020), and Habibi (El-Haj, 2020). The dialect identification task involves three dialect classification levels. These are the binary-level (i.e., MSA vs. DA), region-level (four regions), and country-level (21 countries). (4) Claim Prediction: we use ANS-claim (Khouja, 2020), which is a factuality prediction of claims corpus created using the credibility of the editors as a proxy for veracity (true/false). (5) Machine Generation: for machine generated text detection (i.e, machine vs. human), we use the machine manipulated version of AraNews dataset (Nagoudi et al., 2020). To create this dataset, a list of words are selected (based on their POS tags) and substituted by a chosen word from the k-most similar words in an Arabic word embedding model. Structured Prediction. This task cluster includes two tasks: (1) Arabic NER: we consider two publicly available Arabic NER datasets, ANERcorp (Benajiba and Rosso, 2007) and AQMAR (Schneider et al., 2012). (2) Arabic POS Tagging: we use two POS Tagging datasets, the multidialect Arabic POS dataset Darwish et al. (2018) and the Arabic POS tagging part of XGLUE (Liang et al., 2020). Text Classification. In this task cluster, we explore topic classification employing three documentlevel classification datasets: Khaleej (Abbas et al., 2011), Arabic News Text (ANT) (Chouigui et al., 2017), and OSAC (Saad and Ashour, 2010). Semantic Textual Similarity. This cluster aims to measure the semantic relationship between a pair of sentences. For this, we use the (1) STS re-
6
gression: data from Ar-SemEval-2017 (Cer et al., 2017) (which is a set of Arabic sentence pairs each labeled with a numerical score from the interval [0..1] indicating the degree of semantic similarity). We also use (2) STS classification where a pair of questions is assumed to be semantically similar if they have the same exact meaning and answer. We use the semantic question similarity in Arabic dataset (Q2Q) proposed by Seelawi et al. (2019) where each pair is tagged with “1" (question has the same meaning and answer) or “0" (not similar). Natural Language Inference. This cluster covers the following two tasks: (1) Arabic NLI: we use the Arabic part of the cross-lingual natural language inference (XNLI) corpus (Conneau et al., 2018). The goal is determining whether a text (hypothesis) is false (contradiction), undetermined (neutral), or true (entailment), given a another text (premise). (2) Fact-checking: in order to build a fact-checking benchmark component, we use Unified-FC (Baly et al., 2018) and ANS (Khouja, 2020). Both of these datasets target stance and factuality prediction of claims from news and social media. The two datasets are manually created by annotating the stance between a claim-document pair with labels from the set {agree, disagree, discuss, unrelated}. Word Sense Disambiguation. We use the Arabic WSD benchmark (El-Razzaz et al., 2021), a context-gloss pair dataset extracted from an MSA dictionary. It consists of 15k senses for 5k unique words with an average of three senses for each word. Question Answering. We concatenate four Arabic and multilingual QA datasets. These are ARCD (Mozannar et al., 2019), MLQA (Lewis et al., 2020b), TyDi QA (Artetxe et al., 2020), and XQuAD (Artetxe et al., 2020).

heading: 4 Language Models
text: In this section, we list multilingual PLMs that include Arabic in its coverage only by name, for space but provide a description of each of them in Appendix A. Multilingual LMs. These are mBERT (Devlin et al., 2019) , XLM-R (Conneau et al., 2020b), GigaBERT (Lan et al., 2020), and mT5 (Xue et al., 2021). Arabic LMs. These are AraBERT (Antoun et al., 2020), ArabicBERT (Safaya et al., 2020), ArabicALBERT (Safaya, 2020), QARiB Chowdhury et al. (2020), ARBERT & MARBERT (Abdul-Mageed
et al., 2021), CamelBERT (Inoue et al., 2021), JABER and SABER (Ghaddar et al., 2021), and AraT5 (Nagoudi et al., 2022). Table A.1 (Appendix A) shows a comparison between the multilingual as well as the Arabic PLMs in terms of (1) training data size, (2) vocabulary size, (3) language varieties, and (4) model configuration and architecture.

heading: 5 Model Evaluation on ORCA
text: This section shows experimental settings and performance of 18 multilingual and Arabic language models on ORCA downstream tasks.6 Baselines. For comparison, we finetune the multilingual language models mBERT and XLM-RBase on all training data of ORCA benchmark. Evaluation. For all models and baselines, across all tasks, we identify the best model on the respective development data split (Dev) and blind-test on the testing split (Test). We methodically evaluate each task cluster, ultimately reporting a single ORCA score following Wang et al. (2018); AbdulMageed et al. (2021). ORCA score is simply the macro-average of the different scores across all tasks and task clusters, where each task is weighted equally. We compute the ORCA score for all 18 language models. Results. We present results of all language models and the baselines on each task cluster of ORCA independently using the relevant metric, for both Dev (see Table C.1 in Appendix C) and Test (see Table 4). As Table 4 shows, ARBERTv2 (M3) achieves the highest ORCA score across all the tasks and also for MSA tasks only7 (with ORCA score=74.04 and Avg. MSA score=75.13), followed by CamelBERTmsa (M11) in both cases with 73.35 and 73.64, respectively. Regarding the dialect tasks, we note that MARBERTv2 (M4) achieves the best dialect ORCA score (Avg. DA score=74.62) followed by QARIB (M9) with 74.47. We also note that AraELECTRA (M8) achieves the best results in six individual tasks out of 26, followed by MARBERTv2 (M4) which excels in five individual tasks. Analysis. As our experiments imply, ORCA allow us to derive unique insights. Example insights that
6We exclude JABER and SABER (Ghaddar et al., 2021) from the evaluation as these are not supported by the Transformers library.
7We consider a task an MSA task if it has more than 98% of samples predicted as MSA using the MSA vs. DA classifier (see Table 3).
7
lation. B1, B2: Two baselines mBERT (Devlin et al., 2019) and XLM-R (Liu et al., 2019a). M1, M2: ARBERT, MAR-
BERT (Abdul-Mageed et al., 2021). M3, M4: ARBERTV2 and MARBERTV2. M5, M6, M7, and M8: AraBERTv1[v2, tw],
and AraElectra (Antoun et al., 2020, 2021). M9: QARiB (Chowdhury et al., 2020) M10, M11, M12, and M13:
CamelBERTmix[msa, da, ca] (Inoue et al., 2021). M14: GigaBERTv4 (Chowdhury et al., 2020). M15: Arabic BERT (Chowdhury et al., 2020). M16 : Arabic Albert (Lan et al., 2020). Avg. DIA and Avg. MSA: The average of dialect and MSA tasks. ORCAscore : Average overall Dia and MSA tasks. ⋆MSA tasks. ‡DIA tasks. A task is considered as an MSA if it has more than 98% samples predicted as MSA using an MSA vs. DIA classifier (see Table 3).
can be derived from Table 4 are: (a) a model such as M6 (i.e., AraBERTv2) that is pretrained with historical data (AlSafeer newspaper) would excel on old datasets (e.g., TC, QA, and WSD); while M4 (i.e., MARBERTv2) excels especially on datasets from social media since it is pretrained with a large Twitter collection. In addition, since ORCA arranges the metrics into one dedicated to dialect, another to MSA, and a third to both (ORCA score), it is much easier to compare model performance across the DA-MSA dimensions.

heading: 6 Analysis of Model Computational Cost
text: We also compare the Arabic language models in terms of computational cost using the average time needed for convergence (in minutes) and average
number of epochs to convergence as identified on Dev sets. For this, we finetune all models for a maximum of 25 epochs on all ORCA tasks. We report results in terms of average of three runs. Figure E.2 (Appendix E) shows for each model the total time needed for convergence (out of 25 epochs), and Figure E.1 (Appendix E) shows average convergence time and average number of epochs till convergence. As E.2 (Appendix E) shows, Arabic Albert is the fastest model (52.26 min) to finetune for 25 epochs, but it achieves the lowest ORCA score. Excluding Arabic Albert, we observe a near constant time (between 60.32-63.69 mins) for all other models. Among the top five models, as Figure E.1 (Appendix E) shows, we also observe that ARBERTv1 is the fastest (in terms
8
of average convergence time and number of epochs needed to converge) and is followed by QARiB.

heading: 7 Conclusion
text: We presented ORCA, a large and diverse benchmark for Arabic natural language understating tasks composed of 60 datasets that are arranged in seven task clusters. To facilitate future research and adoption of our benchmark, we offer a publicly-available interactive leaderboard with a useful suite of tools and extensive meta-data. In addition, we provide a comprehensive and methodical evaluation as well as meaningful comparisons between 18 multilingual and Arabic language models on ORCA. We also compare the models in terms of computing needs. As our results show, ORCA is challenging and we hope it will help standardize comparisons and accelerate progress both for Arabic and multilingual NLP.

heading: 8 Limitations
text: We identify the following limitations:
1. Although we strive to include tasks in all Arabic varieties, available downstream datasets from certain countries such as Mauritania and Djibouti are almost nonexistent and so are not covered in ORCA. In addition, there is a need in the community to create more datasets for several Arabic dialects. This includes, for example, dialects such as Iraqi, Sudanese, and Yemeni. With the introduction of more datasets for such dialects, ORCA’s coverage can be further extended. Regardless, as Figure F.1 (Appendix F) shows, ORCA datasets are quite diverse from a geographical perspective.
2. Although ORCA currently covers both dialectal Arabic (DA) and MSA, it does not pay as much attention to the classical variety of Arabic (CA) due to historical reasons. That is, the community did not invest as much efforts creating and releasing datasets involving CA. However, as more unlabeled datasets become available and with an undergoing positive change in the culture around data sharing, this is likely to change in the near future. Again, this will make it possible to extend ORCA to better cover CA in the future.
3. Although benchmarks in general are useful in encouraging standardize evaluations and
meaningful comparisons, and can help motivate progress within the community, they also run the risk of contributing to a culture of leaderboard chasing that is not necessarily useful. That is, although scientific research advances due to competition, it also thrives through partnerships and collaborations that bring the best from diverse groups. It is in the context of this collaborative culture that we hope ORCA will be perceived and used.

heading: 9 Ethics Statement and Broad Impact
text: Encouraging standardized evaluations and contributing to a collaborative research culture. Similar to some other research communities, progress in the Arabic NLP community has been hampered for a long time by absence of standardized and meaningful evaluations for some tasks. This is due to several reasons, including the culture around data sharing but also as a result of insufficient funding and lack of strong training programs. This has made it challenging to measure progress. The Arabic NLP community is now expanding, and a culture of collaboration is being built as part of the larger positive developments within the overall NLP community itself. As such, it is now ripe time to introduce benchmarks that can help this ongoing progress. We hope there will be wide adoption of ORCA and that our work will trigger more efforts to create more benchmarks, including for newer tasks in what could be a virtuous cycle.
Data privacy. Regarding data involved in ORCA, we develop the benchmark using data from the public domain. For this reason, we do not have serious concerns about privacy.
Sufficient assignment of credit to individual data sources. Another important consideration in benchmarking is how credit is assigned to creators of the individual datasets. To ensure sufficient credit assignment, we refer users to the original publications, websites, GitHub repositories where a dataset originated and link all these sources in our leaderboard. We also provide bibliographic entries for all these sources that users can easily copy and paste in order to cite these original sources. By encouraging citation of original sources in any publications in the context of ORCA use, we hope to afford additional visibility to many of the individual datasets.
9

heading: Acknowledgements
text: MAM gratefully acknowledges support from Canada Research Chairs (CRC), the Natural Sciences and Engineering Research Council of Canada (NSERC; RGPIN-2018-04267), the Social Sciences and Humanities Research Council of Canada (SSHRC; 435-2018-0576; 895-2020-1004; 895- 2021-1008), Canadian Foundation for Innovation (CFI; 37771), Digital Research Alliance of Canada (DRAG),8 UBC ARC-Sockeye,9 Advanced Micro Devices, Inc. (AMD), and Google. Any opinions, conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of CRC, NSERC, SSHRC, CFI, DRAG, AMD, Google, or UBC ARC-Sockeye.

heading: A Language Models
text: In this section, we provide a description of the multilingual MLM that include Arabic in its coverage.
A.1 Multilingual LMs mBERT is the multilingual version of BERT (Devlin et al., 2019) which is a multi-layer bidirectional encoder representations from Transformers (Vaswani et al., 2017) trained with a masked language modeling. Devlin et al. (2019) present two architectures: Base and Large. BERT models were trained on English Wikipedia10 and BooksCorpus (Zhu et al., 2015). mBERT is trained on Wikipedia for 104 languages (including ∼ 153M Arabic tokens). XLM-R (Conneau et al., 2020b) is a transformerbased multilingual masked language model pretrained on more than 2TB of filtered CommonCrawl data in 100 languages, including Arabic (2.9B tokens). XLM-R uses a Transformer model trained a multilingual version of masked language modeling of XLM (Conneau and Lample, 2019). XLM-R comes with two sizes and architectures: Base and Large. The XLM-RBase architecture contains 12 layers, 12 attention heads, 768 hidden units, and 270M parameters. The XLM-RLarge architecture has 124 layers, 16 attention heads, 1024 hidden units, and 550M parameters. While both XLM-R models use the same masking objective as BERT, they do not include the next sentence prediction objective used in BERT. GigaBERT (Lan et al., 2020) is a customized bilingual BERT-based model for Arabic and English pretrained on a corpus of 10B tokens collected from different sources, including: English and Arabic Gigaword corpora,11, OSCAR (Suárez et al., 2019), and Wikipedia. GigaBERT is designed specifically for zero-shot transfer learning from English to Arabic on information extraction tasks. mT5 (Xue et al., 2021) is the multilingual version of Text-to-Text Transfer Transformer model (T5) (Raffel et al., 2020). The T5 model architecture is
10https://www.wikipedia.org/ 11https://catalog.ldc.upenn.edu/LDC2011T07
17
Byte Pair Encoding (Wei et al., 2021). ARBERTv2: a new model proposed in this paper.
essentially an encoder-decoder Transformer similar in configuration and size to BERTBase. The T5 model treats every text-based language task as a “text-to-text" problem, (i.e. taking text format as input and producing new text format as output), where multi-task learning is applied with several NLP tasks: question answering, document summarization, machine translation, and sentiment classification. mT5 is trained on the “Multilingual Colossal Clean Crawled Corpus" (or mC4 for short), which is ∼ 26.76TB for 101 languages (including Arabic with more than ∼ 57B tokens) generated from 71 Common Crawl dumps.
A.2 Arabic LMs
Several Arabic LMs have been developed. We describe the most notable among these here.
AraBERT (Antoun et al., 2020) is the first pretrained language model proposed for Arabic. It is based on the two BERTBase and BERTLarge architectures. AraBERTBase (Antoun et al., 2020) is trained on 24GB of Arabic text (70M sentences and 3B tokens) collected from Arabic Wikipedia, Arabic news, Open Source International dataset (OSIAN) (Zeroual et al., 2019), and 1.5B words corpus from (El-Khair, 2016). In order to train BERTLarge Antoun et al. (2021) use the same
AraBERTBase data augmented with the unshuffled Arabic OSCAR dataset (Suárez et al., 2019) and news articles provided by As-Safir newspaper12 (77GB or 8.8B tokens) . The augmented data is also used to train AraELECTRALarge–an Arabic language model that employs an ELECTRA objective (Clark et al., 2020). ArabicBERT is an Arabic BERT-based model proposed by Safaya et al. (2020) Authors pretrain fourr variants: ArabicBERTMini, ArabicBERTMedium, ArabicBERTBase, ArabicBERTLarge.13 The models are pretrained on unshuffled Arabic OSCAR (Suárez et al., 2019), Arabic Wikipedia, and other Arabic resources which sum up to 95GB of text (∼ 8.2B tokens). Arabic-ALBERT (Safaya, 2020) is an Arabic language representation model based on A Lite Bert (ALBERT) (Lan et al., 2019). ALBERT is a Transformer-based neural network architecture (similar to BERT and XLM-R) with two parameter reduction techniques proposed to increase the training speed and lower memory consumption of the BERT model. Arabic-ALBERT is pretrained on ∼ 4.4B tokens extracted from Arabic OSCAR (Suárez et al., 2019) and Arabic Wikipedia. Arabic-ALBERT comes with three dif-
12https://www.assafir.com/ 13https://github.com/alisafaya/Arabic-BERT
18
ferent architectures: Arabic-ALBERTBase, ArabicALBERTLarge, Arabic-ALBERTXLarge. QARiB. Chowdhury et al. (2020) propose the QCRI ARabic and Dialectal BERT (QARiB) model. QARiB is trained on a collection of 97GB of Arabic Text (14B tokens) on both MSA (180 Million sentences) and Twitter data (420 Million tweets). Authors use the Twitter API to collect Arabic tweets, keeping only tweets identified as Arabic by Twitter language filter. For MSA data in QARiB is a combination of Arabic Gigaword,14, Abulkhair Arabic Corpus (El-Khair, 2016), and OPUS (Tiedemann, 2012). ARBERT (Abdul-Mageed et al., 2021) is a pretrained language model focused on MSA. ARBERT is trained using the same architecture as BERTBase with a vocabulary of 100K WordPieces, making ∼ 163M parameters. ARBERT exploits a collection of Arabic datasets comprising 61GB of text (6.2B tokens) from the following sources: El-Khair El-Khair (2016), Arabic Gigaword,15, OSCAR (Suárez et al., 2019), OSIAN (Zeroual et al., 2019), Arabic Wikipedia, and Hindawi Books.16
ARBERTv2. We provide a new Arabic version of ARBERT, by further pretraining ARBERT on 243GB MSA dataset (70GB MSA data from various sources and 173GB extracted and cleaned from the Arabic part of the multilingual Colossal Clean Crawled Corpus (mC4) (Xue et al., 2021). MARBERT (Abdul-Mageed et al., 2021) is a pretrained language model focused on both dialectal Arabic and MSA. This model is trained on a sample of 1B Arabic tweets (128GB of text, 15.6B tokens). In this dataset, authors keep only tweets with at least 3 Arabic words (based on character string matching) regardless of whether the tweet has non-Arabic string or not. MARBERT uses the same vocabulary size (100K WordPieces) and network architecture as ARBERT (BERTBase), but without the next sentence prediction objective since tweets are short. MARBERTv2. Abdul-Mageed et al. (2021) further pretrain MARBERT with additional data using a larger sequence length of 512 tokens for 40 epochs. CamelBERT (Inoue et al., 2021) is pre-trained using BERTBase architecture on four types of Arabic datasets: MSA (107GB), dialectal Arabic (54GB), classical Arabic (6GB), and a mixture of the last
14https://catalog.ldc.upenn.edu/LDC2011T11 15https://catalog.ldc.upenn.edu/LDC2009T30 16https://www.hindawi.org/books
three datasets (167GB). CamelBERT is trained using a small vocabulary of 30K tokens (in WordPieces). JABER and SABER (Ghaddar et al., 2021) are BERT-based models (Base and Large) pretraind on 115GB of text data collected from Common Crawl (CC), OSCAR (Suárez et al., 2019), OSIAN (Zeroual et al., 2019), El-Khair El-Khair (2016), and Arabic Wikipedia. In order to overcome the outof-vocabulary problem and improve the representations of rare words, JABER is trained using a Byte-level Byte Pair Encoding (BBPE) (Wei et al., 2021) tokenizer with a vocabulary size of 64K. AraT5 (Nagoudi et al., 2022) is an Arabic textto-text Transformer model dedicated to MSA and dialects. It is essentially an encoder-decoder Transformer similar in configuration and size to T5 (Raffel et al., 2020). AraT5 is trained on more than 248GB of Arabic text (70GB MSA and 178GB tweets), where the data is from the following sources: AraNews (Nagoudi et al., 2020), El-Khair El-Khair (2016), Gigaword,17, OSCAR (Suárez et al., 2019), OSIAN (Zeroual et al., 2019), Wikipedia Arabic, and Hindawi Books.18
Table A.1 shows a comparison between the multilingual as well as the Arabic language models in terms of (1) training data size, (2) vocabulary size, (3) language varieties, and (4) model configuration and architecture.
B X-Specific Benchmarks
CLUE. Xu et al. (2020) introduce CLUE, a benchmark for Chinese NLU. It covers nine tasks spanning single-sentence/sentence-pair classification, text classification, coreference resolution, semantic similarity, and question answering. FLUE. Le et al. (2020) offer FLUE, a French NLU benchmark involving six datasets with different levels of difficulty, degrees of formality, and domains. FLUE is arranged into three tasks: text classification, paraphrasing, and NLI. IndoNLU. Wilie et al. (2020) present IndoNLU, a benchmark for Bahasa Indonesian NLU with 12 downstream tasks organized into five task clusters: sentence classification, structure protection, text classification, semantic similarity, and question answering. JGLUE. Kurihara et al. (2022) propose JGLUE, a Japanese NLU benchmark consisting of six
17https://catalog.ldc.upenn.edu/LDC2009T30 18https://www.hindawi.org/books
19
datasets arranged into three task clusters: sentence classification, text classification, and question answering. KorNLI and KorSTS. Ham et al. (2020) release KorNLI and KorSTS, two benchmark datasets for NLI and STS in the Korean language.

heading: C ORCA Evaluation
text: In this section, we provide additional information about the evaluation as follows:
• Performance of Arabic BERT-based models on ORCA Dev splits are shown in Table C.1.
• Figure C.1 shows ORCA scores from the different PLMs across all 29 tasks in ORCA benchmark.
• Figure C.2 shows models’ ORCA scores across all clusters in ORCA benchmark.
• Figure C.3 shows models’ F1 scores across all tasks in sentence classification cluster.

heading: D Public leaderboard.
text: In this section, we provide additional screenshots for ORCAleaderboard, as follows:
• Figure D.1 shows an example of tasks sorted alphabetically.
• Figure D.3 shows detailed scores by all models for a given task..
• Figure D.3 shows detailed information about each task cluster and associated tasks, with each task assigned an identifier, language variety, evaluation metric, a link to the dataset website/GitHub/paper and bibliographic information.

heading: E Analysis of Model Computational Cost
text: In this section, we provide additional information about the models’ computational cost, as follows:
• Figure E.1 shows the average number of epochs (in orange), and time needed to converge (mins, in blue) for all the studied pretrained language models across all ORCA tasks.
• Figure E.2 shows the time needed in minutes to fine-tune (25 epochs). We compute the average time of three runs across all ORCA tasks.

heading: F ORCA Data
text: In this section, we provide additional information about ORCAData, as follows:
• Table F.1 shows a randomly picked examples from the dialectal portion of ORCA Train datasets.
• Figure F.1 shows the predicted country-level distribution, in percentage, in the dialectal portion of ORCA.
20
B1, B2: Two baselines mBERT (Devlin et al., 2019) and XLM-R (Liu et al., 2019a). M1, M2: ARBERT, MARBERT (Abdul-
Mageed et al., 2021). M3, M4: ARBERTV2 and MARBERTV2. M5, M6, M7, and M8: AraBERTv1[v2, tw], and AraElec-
tra (Antoun et al., 2020, 2021). M9: Qraib (Chowdhury et al., 2020) M10, M11, M12, and M13: CamelBERTmix[msa, da, ca] (In-
oue et al., 2021). M14: GigaBERTv4 (Chowdhury et al., 2020). M15: Arabic BERT (Chowdhury et al., 2020). M16 : Arabic Albert (Lan et al., 2020). Avg. Dia, and Avg. MSA: The average of dialect and MSA tasks. ORCAscore : Average overall Dia and MSA tasks. ⋆MSA tasks. ‡DIA tasks. A task is considered as an MSA if it has more than 98% samples predicted as MSA using an MSA Vs DIA classifier (see Table 3).
21
22
23
24
Figure E.1: The average number of epochs (in orange), and time needed to converge (mins, in blue) for all the studied pretrained language models across all ORCA tasks.
Country Example Dataset Label Egypt . ú GAª̄ñ K @ñJ. J k @Q Êm. ' @ ð AJ ËA¢ @ Emotion Happy èñ¢ k éJ ̄ Öß ÕËð @Q º ® K ÈAJ Ó B@ ¬B @ éJ ̄ Iª¢̄ QÓ @ á« HYj J K I K A ̄ , ½Òê ®K áË Adult Not Adult
K Qî DË @ AêÊëB I. J j. K. é m ñË@ é ®Ê mÌ'@ Sarcasm Sarcasm
Jordan
Ë@ 20 A K @ é ®J ª ¼YªK. . Pñº ©Ôg. @ Pñ ̄ @ . àA « XAë éË Q K @ Ð PB àðñK @ Gender Male
A JJ Ê« J.¢ K hP Ð@ AK éJ Ò AK ú » @Pð áÓð Offensive Not Offensive
jm.Ì'Aë ¡ . QK Yg@ AÓ Abusive Abusive
KSA úæ áK ñ @ X @ Dangerous Dangerous
! ÕºK. ñÊ̄ úÎ« ÕºK YK @ àñ¢m 'ð ú GA «B@ àñ ̄ Ym ' ÕºK @P ð Emotion Happy Õç' A JË @ I. ̄ @Q K é«A 42 ½J m. 'ð éJ ÓAm× AîD ® K É¿ñ Kð K. A Ó Ñ«X é ̄Qå Ó Q ú Í@ èQ» @ Age Under 25
Kuwait @Q ® . Bð é J J ªË@ð éK PAÒªËAK. Bð H. @QK YK. B @Yg. é ª K éºJ. Ë@  CË Sentiment Negative ©ËA£ èA ®Ë @ K Ā X 5 YªK. é X@ éÖÏð ȬQË @ ð PñJ Ë @ ¡k@ èQÓ É¿ . ú æ
® K Gender Male éë Õº JËAJ « Õæ @ ú Î« ø @ Ê¿ ðñë àAÓ@ éËAÓ X@A« éë Emotion Fear
Table F.1: Randomly picked examples from the dialectal portion of ORCA Train datasets.
25
Figure E.2: The time needed in minutes to fine-tune (25 epochs). We compute the average time of three runs across all ORCA tasks.
Figure F.1: Predicted country-level distribution, in percentage, in the dialectal portion of ORCA.
26
ACL 2023 Responsible NLP Checklist

heading: A For every submission:
text: 3 A1. Did you describe the limitations of your work?
8
A2. Did you discuss any potential risks of your work? Not applicable. Left blank.

heading: 3 A3. Do the abstract and introduction summarize the paper’s main claims?
text: abstract, section 1
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? 3
3 B1. Did you cite the creators of artifacts you used? 3
3 B2. Did you discuss the license or terms for use and / or distribution of any artifacts? 9

heading: 3 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided
text: that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? 3, 4
7 B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Since the data belong to the public domain, we do not have serious concerns about privacy or anti-social language beyond what already exists online and is accessible to anyone.

heading: 3 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and
text: linguistic phenomena, demographic groups represented, etc.? 3,4,5
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. 3,4
C 3 Did you run computational experiments? 6

heading: 3 C1. Did you report the number of parameters in the models used, the total computational budget
text: (e.g., GPU hours), and computing infrastructure used? 6,5
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? No response.

heading: 3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary
text: statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? 5,6

heading: 3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did
text: you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? 5
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? Not applicable. Left blank.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? Not applicable. Left blank.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? Not applicable. Left blank.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? Not applicable. Left blank.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? Not applicable. Left blank.

heading: Future Work
text: However, as more unlabeled datasets become available and with an undergoing positive change in the culture around data sharing, this is likely to change in the near future. Again, this will make it possible to extend ORCA to better cover CA in the future. 3. Although benchmarks in general are useful in encouraging standardize evaluations and
meaningful comparisons, and can help motivate progress within the community, they also run the risk of contributing to a culture of leaderboard chasing that is not necessarily useful. That is, although scientific research advances due to competition, it also thrives through partnerships and collaborations that bring the best from diverse groups. It is in the context of this collaborative culture that we hope ORCA will be perceived and used