Estimating the Level of Dialectness Predicts Interannotator Agreement in Multi-dialect Arabic Datasets (2024)

Abstract: On annotating multi-dialect Arabic datasets, it is common to randomly assign the samples across a pool of native Arabic speakers. Recent analyses recommended routing dialectal samples to native speakers of their respective dialects to build higher-quality datasets. However, automatically identifying the dialect of samples is hard. Moreover, the pool of annotators who are native speakers of specific Arabic dialects might be scarce. Arabic Level of Dialectness (ALDi) was recently introduced as a quantitative variable that measures how sentences diverge from Standard Arabic. On randomly assigning samples to annotators, we hypothesize that samples of higher ALDi scores are harder to label especially if they are written in dialects that the annotators do not speak. We test this by analyzing the relation between ALDi scores and the annotators’ agreement, on 15 public datasets having raw individual sample annotations for various sentence-classification tasks. We find strong evidence supporting our hypothesis for 11 of them. Consequently, we recommend prioritizing routing samples of high ALDi scores to native speakers of each sample’s dialect, for which the dialect could be automatically identified at higher accuracies.

text: Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 778–789 August 11-16, 2024 ©2024 Association for Computational Linguistics
On annotating multi-dialect Arabic datasets, it is common to randomly assign the samples across a pool of native Arabic speakers. Recent analyses recommended routing dialectal samples to native speakers of their respective dialects to build higher-quality datasets. However, automatically identifying the dialect of samples is hard. Moreover, the pool of annotators who are native speakers of specific Arabic dialects might be scarce. Arabic Level of Dialectness (ALDi) was recently introduced as a quantitative variable that measures how sentences diverge from Standard Arabic. On randomly assigning samples to annotators, we hypothesize that samples of higher ALDi scores are harder to label especially if they are written in dialects that the annotators do not speak. We test this by analyzing the relation between ALDi scores and the annotators’ agreement, on 15 public datasets having raw individual sample annotations for various sentence-classification tasks. We find strong evidence supporting our hypothesis for 11 of them. Consequently, we recommend prioritizing routing samples of high ALDi scores to native speakers of each sample’s dialect, for which the dialect could be automatically identified at higher accuracies.

heading: 1 Introduction
text: Arabic is spoken natively by over 420 million people and is an official language of 24 countries (Bergman and Diab, 2022), making it an important language for NLP systems. However, NLP for Arabic faces a major challenge in that usergenerated text is typically a mixture of Modern Standard Arabic (MSA)—the standardized variant that is taught in schools and used in official communications and newspapers—and regional variants of Dialectal Arabic (DA), which are used in everyday communications, including both speech and
social media text (Habash, 2010).1 While MSA can be largely understood by most Arabic speakers, the different variants of DA are not always fully mutually intelligible.
Despite this mutual unintelligibility, a common practice when developing datasets for multi-dialect Arabic NLP is to randomly recruit annotators without regard to their dialect. However, routing dialectal content to speakers of a different dialect for annotation or moderation can present real problems. For example, it has been shown to contribute to unjust online content moderation of DA (Business for Social Responsibility, 2022), and racially biased toxicity annotation in American English varieties (Sap et al., 2022). Two recent studies of multi-dialect DA annotation showed that for annotating hate speech or sarcasm, respectively, annotators were more lenient (for hate speech) and more accurate (for sarcasm) when annotating sentences in their native dialect (Bergman and Diab, 2022; Abu Farha and Magdy, 2022). The authors of both studies made the same recommendation for creating new Arabic datasets, namely to first identify the dialect of each sample and then route it to appropriate annotators.
This recommendation is theoretically appealing, but presents practical difficulties since automatic dialect identification (DI) is challenging (AbdulMageed et al., 2023), and existing systems assume a single correct label when in fact some texts can be natural in different dialects (Keleg and Magdy, 2023; Olsen et al., 2023). Moreover, the representation of native speakers of the different Arabic dialects on crowdsourcing sites might be skewed (Mubarak and Darwish, 2016). Therefore, recruiting native speakers of some Arabic dialects might be challenging, given the tough conditions of the countries in which these dialects are spoken.
1Refer to §B of the Appendix for a further discussion about the relationship between MSA and DA.
778
In this paper, we address these challenges by building on recent work by Keleg et al. (2023), who presented a system for estimating Arabic Level of Dialectness (ALDi)—i.e., the degree to which a sentence diverges from MSA, on a scale from 0 to 1. We hypothesize that as sentences with low ALDi scores do not diverge much from MSA, they can still be understood and accurately annotated by most Arabic speakers, while this will be less true for sentences with high ALDi scores. If our hypothesis holds, then annotation can be made more efficient while maintaining accuracy, by routing samples with low ALDi scores to speakers of any dialect. Only high-ALDi samples need to be routed to native speakers of the appropriate dialect.
We test our hypothesis by investigating the impact of ALDi score on interannotator agreement for 15 publicly released datasets annotated for 6 different sentence-classification tasks.2 We confirm that for most tasks and datasets, higher ALDi scores correlate with lower annotator agreement. A notable exception is the dialect identification (DI) task, where higher ALDi scores correlate with higher agreement, presumably because it is easier to identify a single dialect for sentences that are strongly dialectal. This finding is encouraging for annotation routing, since automatic DI systems may also have higher accuracy on these sentences. We conclude that a combination of automatic ALDi scoring, followed by DI and annotator routing only for high-ALDi sentences, is a promising strategy for annotating multi-dialect Arabic datasets.

heading: 2 Methodology
text: Data We study the impact of ALDi scores on the annotators’ agreement for publicly released Arabic datasets. We analyze datasets satisfying the following criteria:
• Language: Mixture of MSA and DA. • Variation: Targeting multiple variants of DA. • Annotators: Speakers of different variants of
DA that are randomly assigned to the samples. • Tasks Setup: Sentence-level classification. • Released Labels: Individual annotator labels
or the percentage of annotators agreeing on the majority-vote label.3
2Instructions to replicate the experiments can be accessed through https://github.com/AMR-KELEG/ALDi-and-IAA
3For some datasets, the percentage of annotators agreeing on the majority vote is weighted by their performance on the annotation quality-assurance test samples. This distinction is irrelevant to our study, where we only consider whether all annotators agreed or not.
We searched for datasets on Masader, a community-curated catalog of Arabic datasets (Alyafeai et al., 2021; Altaher et al., 2022). Each dataset on Masader has a metadata field for the variants of Arabic included. We discarded the datasets that only included MSA samples, and manually inspected the remaining 151. After identifying 28 potential datasets that satisfy the criteria above, we contacted the authors of the datasets that do not have the individual annotations publicly released. Eventually, we had 15 datasets to analyze, listed in Table 1, covering: Offensive Text Classification, Hate Speech Detection, Sarcasm Detection, Sentiment Analysis, Speech Act Detection, Stance Detection, and Dialect Identification.
Analysis For each dataset, we compute the Arabic Level of Dialectness (ALDi) score for each annotated sample (sentence) using the SentenceALDi model (Keleg et al., 2023), which returns a score from 0 (MSA/non-dialectal) to 1 (strongly dialectal). To investigate the effect of ALDi on annotator agreement, we bin the samples by their ALDi score into 10 bins of width 0.1. We compute % full agree, the percentage of samples in that bin for which all the annotators agreed on a single label. We employ Pearson’s correlation coefficient to analyze the relation between ALDi (represented by each bin’s midpoint ALDi score) and % full agree, and also report the slope of the best-fitting line as a measure of the effect size.4 As aforementioned, our initial hypothesis is that % full agree negatively correlates with high ALDi scores.

heading: 3 Results and Discussion
text: We use scatter plots to visualize the relation between % full agree and ALDi on the studied datasets, as shown in Figure 1. Additionally, the histograms of samples across the different bins indicate the dialectal content within the dataset. As per Table 1, 6 datasets out of the 15 have more than 50% of the samples with ALDi scores less than 0.1, which are expected to be written in MSA. However, we found that the overall trends depicted
4The exact values of the slopes and correlation coefficients depend on the number of bins. However, we got similar qualitative results on using 4 or 20 equal-width bins. 10 bins are enough to check if trends are non-linear while keeping a reasonable number of samples in the smallest bins. We also fitted logistic regression (logreg) models using ALDi as a continuous variable and a binary outcome Full Agreement (Yes/No) for each sample. Both analysis tools reveal similar patterns (See Appendix §C) but the binning method provides useful additional visualization.
in Figure 1 will not be affected if we discard these samples with low ALDi scores and only focus on the rest.
For non-DI tasks, ALDi negatively correlates with agreement. Inspecting the trends depicted in Figure 1, strong negative Pearson’s correlation
coefficients exist for 8 out of the 12 datasets for the non-DI tasks (sentiment analysis, sarcasm, hate speech, and stance detection). Both the trends (quantified by the slope m) and the correlation coefficients for most of the tasks indicate that the percentage of samples for which all the annotators assign the same label decreases as the ALDi scores increase, often by a large margin.5 We notice different trends for DI that we elaborate on below.
For DI, agreement is lowest for mid-range ALDi scores (if MSA is a possible label) or low ALDi scores (if it is not). By definition, MSA sentences have an ALDi of 0, and normally the ALDi estimation model assigns them very low scores.
For the ArSarcasm-v1 and iSarcasm datasets, the set of labels for the DI task includes MSA (i.e., some sentences in these datasets are not dialectal). For both datasets, one notices high percentages of agreement scores for the bin having ALDi scores ∈ [0, 0.1] (generally agreeing that the label is MSA). The percentages decrease for the few succeeding bins, before rising again for the bins with high ALDi scores. Sentences of high ALDi scores (e.g., ∈ [0.8, 1]) are expected to have multiple dialectal cues, which increases the chance of attributing them to a single dialect. For sentences of intermediate ALDi scores, annotators can agree a sentence is not in MSA, however, they would struggle to determine the dialect of the sentence, which is manifested as having lower percentages of the full agreement for these bins.
5Refer to Appendix §E for a possible explanation of the unexpected trends of the ArSAS dataset.
The authors of the DART dataset do not include MSA in the labels since they curated sentences with distinctive dialectal terms. This explains the low percentage of full agreement for the bin of ALDi scores ∈ [0, 0.1], unlike the other two DI datasets. However, the pattern of having higher full agreement percentages for bins with higher ALDi scores still holds.

heading: 4 Analysis of Trends by Class Label
text: A more nuanced analysis of the non-DI datasets can be done by splitting the samples according to their majority-vote labels (See Appendix §D).
The declining trend for agreement as ALDi scores increase is consistently salient for the negative class of the different tasks (i.e., Not Sarcastic, Neutral (sentiment), Non-offensive). One explanation is that agreeing on one of these labels requires fully understanding the sample, which is expected to be harder for non-native speakers of the dialects of sentences with high ALDi scores. Moreover, the general usage of MSA for formal communications, and DA for more personal ones might be biasing the annotators. Further controlled analysis is required to investigate these intuitive explanations.
Conversely, the presence of specific words might be a strong cue for the label of the overall sentence, which might be useful for the positive classes (i.e., Sarcastic, Positive/Negative, Obscene/Offensive). Consequently, annotators might be able to agree on a label for a sentence even if it contains some unintelligible segments. Table 2 shows five examples, in which the annotators can use cues to label the
whole sentence. Consider the first offensive sentence in the Table. The presence of the MSA insult rK A§ (you insect) is enough to guess that the sentence is offensive, even if the remaining segment is not fully intelligible. Lastly, note that agreeing on a label does not imply it is accurate, especially when relying on cues for annotation.

heading: 5 Conclusion and Recommendation
text: Factors such as task subjectivity and vague guidelines could cause disagreement between annotators. For Arabic, we demonstrate that the Arabic level of dialectness of a sentence (ALDi), automatically estimated using the Sentence-ALDi model (Keleg et al., 2023), is an additional overlooked factor.
Analyzing 15 datasets, we find strong evidence of a negative correlation between ALDi and the full annotator agreement scores for 8 of the 12 nonDialect Identification datasets. Moreover, for the 3 Dialect Identification datasets, we find that annotators have higher agreement scores for samples of higher ALDi scores, which by definition would have more dialectal features. The combination of more dialectal features in a sentence is more probable to be distinctive of a specific dialect.
Previous research recommended routing samples to native speakers of the different Arabic dialects for higher annotation quality. Our analysis indicates that a large proportion of 6 datasets are samples with ALDi scores < 0.1, which are expected to be MSA samples that can be routed to speakers of any Arabic dialect. Moreover, the lower agreement scores for samples with high ALDi scores show that extra care should be given to these samples. Dataset creators should prioritize routing high-ALDi samples to native speakers of the dialects of these samples, for which the dialects can be automatically identified at higher accuracy as these samples have more dialectal cues.
Limitations
The trends we report validate our hypothesis. However, more thorough analyses need to be done to understand how ALDi affects each task given its unique nature. Knowing the demographic information about the annotators might have allowed for revealing deeper insights into how speakers of specific Arabic dialects understand samples from other dialects. However, this would have required running a controlled experiment re-annotating the 15 datasets, which we hope future work will attempt.
Another potential extension to this work is to analyze the interannotator disagreement on annotating dialectal data for token-level tasks. To the best of our knowledge, all the publicly available token-level Arabic datasets are built by carefully selecting samples written in specific dialects and recruiting native speakers of each of these dialects to perform the annotation, after closely training them. However, even if a multi-dialect token-level dataset is annotated by randomly assigning the samples to speakers of different dialects, the analysis would require a new model to estimate the level of dialectness on the token level, since the Sentence-ALDi model used here works at the sentence level.
Lastly, we acknowledge that there are multiple reasons for the annotators to disagree, which include the task’s subjectivity, the annotators’ background, and their worldviews (Uma et al., 2021). However, these factors would have less impact on the annotators’ disagreement if a sample is not fully intelligible.

heading: Acknowledgments
text: This work could not have been done without the help of the datasets’ creators who have kindly agreed to share the individual labels for their datasets’ samples. Thanks, Ibrahim Abu Farha, Nora Alturayeif, and Manal Alshehri. We also thank Hamdy Mubarak, Nuha Albadi, Nedjma Ousidhoum, and Hala Mulki for trying to help with finding the individual annotator labels for some of their datasets. Lastly, we appreciate the efforts of the anonymous ARR reviewers, action editors, and area chairs. Thanks for the insightful discussions and valuable suggestions.
This work was supported by the UKRI Centre for Doctoral Training in Natural Language Processing, funded by the UKRI (grant EP/S022481/1) and the University of Edinburgh, School of Informatics.

heading: A Detailed Description of the Datasets
text: We noticed some discrepancies between the number of samples reported in the papers and the number of samples in the corresponding raw datasets. Despite following any filtration steps described in the papers, some of the datasets had more samples than the ones in the publicly released version, as indicated in Table A1. Additionally, the ArSarcasmv1, Mawqif (Stance Task), Mawqif (Sentiment/Sarcasm Tasks), and ASAD had 516, 170, 151, 191
samples with less than 3 annotations respectively, that we decided to discard from our analysis.
Conversely, we decided to discard the MLMA dataset (Ousidhoum et al., 2019) for which the authors shared with us some of the raw annotations files. The number of samples in these files was too small compared to the number of samples in the public dataset with majority-vote labels. We also discarded another dataset, for which there was a significant discrepancy between the released dataset and the paper’s description of the dataset.

heading: B Code-mixing between MSA and DA
text: Researchers distinguish between Modern Standard Arabic (MSA), and Dialectal Arabic (DA) (Habash, 2010). However, MSA and DA do not exist in isolation, and Arabic speakers sometimes code-mix between terms that can be considered to belong to MSA and others considered to be part of a variety of DA. Notably, some terms can be considered to belong to both MSA and a variety of DA, and even using the surrounding context may not be enough for disambiguation (Molina et al., 2016).
Badawi (1973) recognizes five levels of Arabic used in Egypt, that can be categorized according to the amount of code-mixing in addition to the dialectness of the terms/phrases used. The SentenceALDi model, developed by Keleg et al. (2023), estimates the level of dialectness of Arabic sentences, which provides an automatic proxy to distinguish between Arabic sentences according to how they diverge from MSA. We used the Sentence-ALDi model to study the relation between the ALDi score and the agreement between the annotators for 15 Arabic datasets.

heading: C Discussion about the Analysis
text: As described in §2, each dataset’s samples were split into 10 bins of equal width according to their respective ALDi scores. Afterward, the correlation between each bin’s midpoint ALDi score and the percentage of samples having full agreement % full agree was computed. For each bin, % full agree represents the Maximum Likelihood Estimation (MLE) for the probability that all the annotators agree on the same label for the samples of this bin.
Inability to use Interannotator Agreement metrics for some datasets Automated metrics such as Fleiss’ Kappa (Fleiss, 1971) attempt to measure the Interannotator Agreement (IAA) while accounting for the random agreement/disagreement
Dataset Task (# labels) Labels Distribution of Majority-vote Labels Dataset/Paper Discrepancy
Deleted Comments Dataset (DCD) (Mubarak et al., 2017) Offensive (3) Confidence Offensive (80.31%) Clean (17.76%) Obscene (1.58%) No Majority (0.35%)
-
MPOLD (Chowdhury et al., 2020)
Offensive (2) Individual Non-Offensive (83.12%) Offensive (16.88%)
-
YouTube Cyberbullying Offensive (2) Individual Not (61.38%) HateSpeech (38.62%) - (YTCB) (Alakrot et al., 2018)
ASAD (Alharbi et al., 2021) Sentiment (3) Individual Neutral (67.83%) Negative (15.33%) Positive (15.19%) No Majority (1.65%) The authors shared with us the raw annotation file of which we analyze 100,484 samples with three annotations or more, as opposed to the 95,000 in the released dataset.
ArSAS (Elmadany et al., 2018) Sentiment (4) Confidence Negative (35.38%) Neutral (33.45%) Positive (20.51%) No Majority (6.07%)
Mixed (4.59%)
-
Speech Act (6) Confidence Expression (55.07%) Assertion (38.63%) Question (3.32%) No Majority (1.81%)
Request (0.67%) Recommendation (0.31%) Miscellaneous (0.18%)
ArSarcasm-v1 (Abu Farha and Magdy, 2020) Dialect (5) Individual msa (67.56%) egypt (19.37%) No Majority (5.83%) gulf (3.61%) levant
(3.46%) magreb (0.18%) The samples in the raw annotation artifact shared by the authors has 10,641 samples, as opposed to the 10,547 samples in the released dataset. Sarcasm (2) Individual False (84.24%) True (15.7%) No Majority (0.06%) Sentiment (3) Individual neutral (49.45%) negative (32.57%) positive (14.58%) No Majority (3.4%)
Mawqif (Alturayeif et al., 2022)
Sarcasm (2) Individual No (95.97%) Yes (3.78%) No Majority (0.25%) The authors annotated the same samples for sentiment/sarcasm and stance separately. This was done across 8 different annotation jobs (4 each), for which the authors shared the raw annotation files with us. The number of samples in these files is 4,093 for sentiment/sarcasm and 4,079 for stance, of which 3,942 and 3,909 have three or more annotations. The released dataset is reported to have 4,100 samples. Sentiment (3) Individual Positive (41.15%) Negative (31.46%) Neutral (22.68%) No Majority (4.72%) Stance (3) Individual Favor (60.5%) Against (27.65%) None (7.7%) No Majority (4.14%)
iSarcasm’s test set (Abu Farha et al., 2022)
Dialect (5) Individual msa (32.29%) nile (31.36%) gulf (16.5%) No Majority (15.79%) levant
(2.21%) maghreb (1.86%) The dataset having the individual annotator labels is released as an artifact accompanying the following paper (Abu Farha and Magdy, 2022).Sarcasm (2) Individual 0 (82.07%) 1 (17.93%)
DART (Alsarsour et al., 2018) Dialect (5) Proportion GLF (24.27%) EGY (21.69%) IRQ (21.64%) LEV (16.22%) MGH (16.18%)
-
Table A1: A detailed description of the distribution of the majority-vote labels and the data/paper discrepancies in the datasets with individual annotator labels included in our study. Note 1: No Majority means that multiple labels have the same majority number of votes for Individual/Proportion labels, and Confidence < 0.5 otherwise. Note 2: Some of the samples of the ASAD, ArSarcasm-v1, Mawqif datasets have more than 3 annotations, despite the fact the former two are supposed to have only three annotations per sample.
between annotators. In principle, it might be possible to perform a version of our analysis using Fleiss’ Kappa rather than % full agree as the dependent variable. However, computing Fleiss’ Kappa would require knowledge of the individual annotations for each sample. Such annotations are not available for the ArSAS (Sentiment/Speech Act), DART, and DCD datasets as described in Table A1. Since we wanted to include as many datasets as possible, we used % full agree instead.
Logistic regression as an alternative analysis tool Binning the data leads to a loss of analytical information which might impact the results of the analysis, especially if implausible bins’ boundaries
are used (Wainer et al., 2006). Logistic regression with binary outcomes is an alternative analysis that alleviates the limitations of binning. Each sample has a continuous ALDi score as the independent variable, and a binary outcome Full Annotator Agreement (Yes/No). After fitting a logistic regression model to predict the binary outcome, the coefficient of the ALDi variable measures the impact of ALDi on the odds of full agreement. If this coefficient is negative, then the odds of full annotator agreement decrease as the ALDi score increases.
Figure C1 demonstrates the probability of full agreement of each dataset, in addition to the coefficient of the ALDi score with its 95% confi-
dence interval. For the 8 non-DI datasets with CoefALDi < −0.2, the coefficients can be considered to be statistically significant since the confidence interval does not include zero.
Both analysis tools (correlation analysis and logistic regression) achieve similar results. The same 8 non-DI datasets—ASAD, ArSarcasm-v1 (Sentiment/Sarcasm), Mawqif (Sentiment/Sarcasm/Stance), iSarcasm, and YTCB—have significantly strong negative correlation coefficients as in Figure 1, and statistically significant coefficients for the ALDi variable which are less than -0.2. However, binning the data allows for visualizing the % full agreement as a scatter plot, which can reveal whether the relation between ALDi and the agreement is linear or not, in addition to having a visual way for determining how well the best-fitting line models the data.
Impact of data skewness MSA samples are overrepresented in some of the considered datasets. However, this is generally unproblematic for the analysis, so we opted not to discard the MSA samples. For the method described in Section 2, the samples of each bin are independently used to estimate the MLE of full agreement between annotators. Therefore, the over-representation of MSA samples in some datasets does not impact our analysis.

heading: D Trends by Class Label
text: As mentioned in §4, Figures D2, D3, D4, D5, and D6 visualize the impact of ALDi on the annotator agreement after splitting the samples according to their majority-vote labels. We acknowledge that the number of samples in the bins for some classes is not enough to draw concrete conclusions (e.g., samples with high ALDi scores for the Neutral class of the ArSAS, and Mawqif datasets as per Figure D3).

heading: E The Rising Trend of ArSAS
text: The ArSAS dataset stands out as a dataset with a rising trend for the Speech Act Detection task and a falling trend for the Sentiment Analysis task. Samples of ArSAS were jointly annotated for their sentiment and speech act. Despite having 6 different speech acts, which would arguably make speech act detection harder than sentiment analysis, the Assertion and Expression classes represent 95% of the samples. Looking at their respective trends
shown in Figure D5, the two acts show two different behaviors. Most of the assertive samples have ALDi scores <0.2 (arguably, all are MSA ones). Moreover, the number of Assestion samples with high ALDi scores is not enough to estimate the % full agree for their respective bins. Conversely, the Expression act shows higher agreement as the ALDi score increases.
The creators of ArSAS noticed that most of the Assertion samples were annotated as Neutral, while most of the Expression samples had polarized sentiment (mostly Negative). The annotators might have treated the Assertion class as the act for Objective sentences, while treating Expression as the act for Subjective sentences. This is arguably easier than sentiment analysis which might explain why annotators agree more on the Speech Act label than the Sentiment label for the ArSAS dataset. Further analysis is required to explain the trends of this dataset.
Sentiment Analysis
0.2 0.4 0.6 0.8ALDi 0
0.25 0.50 0.75
1
P (F
ul l A
gr ee
)
0.71 0.47
ASAD
-1.1 -1.0 -4 2
Coef of ALDi = -1.03 *
0.2 0.4 0.6 0.8ALDi 0
0.25 0.50 0.75 1 P (F ul l A gr
ee )
0.53 0.48
ArSarcasm-v1
-0.4 -0.1 -4 2
Coef of ALDi = -0.23 *
0.2 0.4 0.6 0.8ALDi 0
0.25 0.50 0.75
1
P (F
ul l A
gr ee
)
0.49
0.23
Mawqif
-1.4 -1.0 -4 2
Coef of ALDi = -1.2 *
0.2 0.4 0.6 0.8ALDi 0
0.25 0.50 0.75
1
P (F
ul l A
gr ee
)
0.55 0.52
ArSAS
-0.2 -0.1 -4 2
Coef of ALDi = -0.14 *
Sarcasm Detection
0.2 0.4 0.6 0.8ALDi 0
0.25 0.50 0.75
1
P (F
ul l A
gr ee
)
0.74
0.36
iSarcasm
-1.9 -1.3 -4 2
Coef of ALDi = -1.63 *
0.2 0.4 0.6 0.8ALDi 0
0.25 0.50 0.75
1
P (F
ul l A
gr ee
)
0.78
0.5
ArSarcasm-v1
-1.4 -1.1 -4 2
Coef of ALDi = -1.24 *
0.2 0.4 0.6 0.8ALDi 0
0.25 0.50 0.75 1 P (F ul l A gr
ee )
0.95 0.44
Mawqif
-3.4 -2.8 -4 2
Coef of ALDi = -3.1 *
Speech Act Detection
0.2 0.4 0.6 0.8ALDi 0
0.25 0.50 0.75
1
P (F
ul l A
gr ee
)
0.68 0.71
ArSAS
0.0 0.2 -4 2
Coef of ALDi = 0.14 *
Offensive Text Classification and Hate Speech Detection
0.2 0.4 0.6 0.8ALDi 0
0.25 0.50 0.75
1
P (F
ul l A
gr ee
)
0.6 0.63
DCD
0.1 0.2 -4 2
Coef of ALDi = 0.14 *
0.2 0.4 0.6 0.8ALDi 0
0.25 0.50 0.75
1
P (F
ul l A
gr ee
)
0.76 0.82 MPOLD
0.1 0.6 -4 2
Coef of ALDi = 0.34 *
0.2 0.4 0.6 0.8ALDi 0
0.25 0.50 0.75
1
P (F
ul l A
gr ee
) 0.78 0.66
YTCB
-0.7 -0.5 -4 2
Coef of ALDi = -0.61 *
Stance Detection
0.2 0.4 0.6 0.8ALDi 0
0.25 0.50 0.75
1
P (F
ul l A
gr ee
) 0.68
0.43
Mawqif
-1.3 -0.8 -4 2
Coef of ALDi = -1.04 *
Arabic Dialect Identification
0.2 0.4 0.6 0.8ALDi 0
0.25 0.50 0.75
1
P (F
ul l A
gr ee
)
0.12 0.27
iSarcasm
0.6 1.4 -4 2
Coef of ALDi = 0.99 *
0.2 0.4 0.6 0.8ALDi 0
0.25 0.50 0.75
1
P (F
ul l A
gr ee
)
0.56 0.34
ArSarcasm-v1
-1.0 -0.8 -4 2
Coef of ALDi = -0.91 *
0.2 0.4 0.6 0.8ALDi 0
0.25 0.50 0.75
1
P (F
ul l A
gr ee
)
0.48 0.82
DART
1.5 1.8 -4 2
Coef of ALDi = 1.63 *
Figure C1: For each dataset, plots show the estimated probability of full agreement according to each dataset’s fitted logistic regression model. Under each plot, the coefficient of ALDi with its 95% confidence interval is visualized. Nearly all datasets (marked with *) have confidence intervals that do not include zero, meaning the effect of ALDi is statistically significant at p < 0.05. Negative coefficients indicate that higher ALDi scores predict lower agreement.
0.2 0.4 0.6 0.8ALDi 0
20 40 60 80
100
% fu
ll ag
re e
m = -23.9 252
5306
ArSarcasm-v1 (False), = -0.87 *
0.2 0.4 0.6 0.8ALDi 0
20 40 60 80
100
% fu
ll ag
re e
m = 16.47
78
506
ArSarcasm-v1 (True), = 0.62
0.2 0.4 0.6 0.8ALDi 0
20 40 60 80 100 % fu ll ag re
e
m = -31.38 42
2274
Mawqif (No), = -0.97 *
0.2 0.4 0.6 0.8ALDi 0
20 40 60 80
100
% fu
ll ag
re e
m = -6.71
6
23
Mawqif (Yes), = -0.16
0.2 0.4 0.6 0.8ALDi 0
20 40 60 80
100
% fu
ll ag
re e
m = -34.71 47
402
iSarcasm (0), = -0.8 *
0.2 0.4 0.6 0.8ALDi 0
20 40 60 80
100
% fu
ll ag
re e
m = 15.25
10
66
iSarcasm (1), = 0.46
Figure D2: The trends for the classes of the Saracasm Detection datasets. Statistically significant correlation coefficients (ρ) are marked with *.
0.2 0.4 0.6 0.8ALDi 0
20 40 60 80
100
% fu
ll ag
re e
m = 10.38
663
3385
ASAD (Negative), = 0.91 *
0.20.40.60.8ALDi 0
20 40 60 80
100
% fu
ll ag
re e
m = -32.66 3679
29303
ASAD (Neutral), = -0.98 *
0.2 0.4 0.6 0.8ALDi 0
20 40 60 80
100
% fu
ll ag
re e
m = 13.57 935
3908
ASAD (Positive), = 0.82 *
0.2 0.4 0.6 0.8ALDi 0
20 40 60 80
100
% fu
ll ag
re e
m = -4.59
52
408
ArSAS (Mixed), = -0.35
0.2 0.4 0.6 0.8ALDi 0
20 40 60 80
100
% fu
ll ag
re e
m = 7.83 392
2924
ArSAS (Negative), = 0.76 *
0.2 0.4 0.6 0.8ALDi 0
20 40 60 80
100
% fu
ll ag
re e
m = -48.66
29
5873
ArSAS (Neutral), = -0.89 *
0.2 0.4 0.6 0.8ALDi 0
20 40 60 80
100
% fu
ll ag
re e
m = 2.82 169
2272
ArSAS (Positive), = 0.17
0.2 0.4 0.6 0.8ALDi 0
20 40 60 80
100
% fu
ll ag
re e
m = 9.8 133
1618
ArSarcasm-v1 (negative), = 0.65 *
0.2 0.4 0.6 0.8ALDi 0
20 40 60 80
100
% fu
ll ag
re e
m = -31.2
97
3344
ArSarcasm-v1 (neutral), = -0.85 *
0.2 0.4 0.6 0.8ALDi 0
20 40 60 80
100
% fu
ll ag
re e
m = 3.35
62
702
ArSarcasm-v1 (positive), = 0.16
0.2 0.4 0.6 0.8ALDi 0
20 40 60 80
100
% fu
ll ag
re e
m = -9.5
32
458
Mawqif (Negative), = -0.69 *
0.2 0.4 0.6 0.8ALDi 0
20 40 60 80
100
% fu
ll ag
re e
m = -17.17
17
500
Mawqif (Neutral), = -0.55
0.2 0.4 0.6 0.8ALDi 0
20 40 60 80
100
% fu
ll ag
re e
m = -63.59
2
1244
Mawqif (Positive), = -0.89 *
Figure D3: The trends for the classes of the Sentiment Analysis datasets. Statistically significant correlation coefficients (ρ) are marked with *.
0.2 0.4 0.6 0.8ALDi 0
20 40 60 80 100 % fu ll ag re
e
m = -13.95 470
1787
YTCB (Not), = -0.93 *
0.2 0.4 0.6 0.8ALDi 0
20 40 60 80
100
% fu
ll ag
re e
m = -0.82 209
1486
YTCB (HateSpeech), = -0.13
0.2 0.4 0.6 0.8ALDi 0
20 40 60 80
100
% fu
ll ag
re e
m = -10.61
86
4143
DCD (Clean), = -0.83 *
0.2 0.4 0.6 0.8ALDi 0
20 40 60 80
100
% fu
ll ag
re e
m = 36.64
29
126
DCD (Obscene), = 0.78 *
0.20.40.60.8ALDi 0
20 40 60 80
100
% fu
ll ag
re e
m = -2.42 565
15505
DCD (Offensive), = -0.32
0.2 0.4 0.6 0.8ALDi 0
20 40 60 80
100
% fu
ll ag
re e
m = 9.13 188
955
MPOLD (Non-Offensive), = 0.65 *
0.2 0.4 0.6 0.8ALDi 0
20 40 60 80
100
% fu
ll ag
re e
m = 6.32 42
158
MPOLD (Offensive), = 0.24
Figure D4: The trends for the classes of the Offensive Text Classification and Hate Speech datasets. Statistically significant correlation coefficients (ρ) are marked with *.
0.2 0.4 0.6 0.8ALDi 0
20 40 60 80
100
% fu
ll ag
re e
m = -55.48
8
6868
ArSAS (Assertion), = -0.81 *
0.2 0.4 0.6 0.8ALDi 0
20 40 60 80
100
% fu
ll ag
re e
m = 13.93 663
4549
ArSAS (Expression), = 0.89 *
Figure D5: The trends for the Assertion and Expression labels of the ArSAS dataset, which represent 95% of the dataset samples. Statistically significant correlation coefficients (ρ) are marked with *.
0.2 0.4 0.6 0.8ALDi 0
20 40 60 80
100
% fu
ll ag
re e
m = -7.52 26
383
Mawqif (Against), = -0.35
0.2 0.4 0.6 0.8ALDi 0
20 40 60 80
100
% fu
ll ag
re e
m = -18.96
7
158
Mawqif (None), = -0.61
0.2 0.4 0.6 0.8ALDi 0
20 40 60 80
100
% fu
ll ag
re e
m = -19.25 22
1652
Mawqif (Favor), = -0.67 *
Figure D6: The trends for the classes of Mawqif’s Stance dataset. Statistically significant correlation coefficients (ρ) are marked with *.

heading: Future Work
text: However, this would have required running a controlled experiment re-annotating the 15 datasets, which we hope future work will attempt. Another potential extension to this work is to analyze the interannotator disagreement on annotating dialectal data for token-level tasks. To the best of our knowledge, all the publicly available token-level Arabic datasets are built by carefully selecting samples written in specific dialects and recruiting native speakers of each of these dialects to perform the annotation, after closely training them. However, even if a multi-dialect token-level dataset is annotated by randomly assigning the samples to speakers of different dialects, the analysis would require a new model to estimate the level of dialectness on the token level, since the Sentence-ALDi model used here works at the sentence level. Lastly, we acknowledge that there are multiple reasons for the annotators to disagree, which include the task’s subjectivity, the annotators’ background, and their worldviews (Uma et al. , 2021). However, these factors would have less impact on the annotators’ disagreement if a sample is not fully intelligible