ALDi: Quantifying the Arabic Level of Dialectness of Text
Authors: Amr Keleg
Sharon Goldwater
Walid Magdy
Venue: EMNLP 2023 Main (2023)
Keywords: Arabic Dialects
Arabic Dialect Identification
Dialectal Variation
Code-switching
Level of Dialectness

Abstract: Transcribed speech and user-generated text in Arabic typically contain a mixture of Modern Standard Arabic (MSA), the standardized language taught in schools, and Dialectal Arabic (DA), used in daily communications. To handle this variation, previous work in Arabic NLP has focused on Dialect Identification (DI) on the sentence or the token level. However, DI treats the task as binary, whereas we argue that Arabic speakers perceive a spectrum of dialectness, which we operationalize at the sentence level as the Arabic Level of Dialectness (ALDi), a continuous linguistic variable. 
We introduce the AOC-ALDi dataset (derived from the AOC dataset), containing 127,835 sentences (17\% from news articles and 83\% from user comments on those articles) which are manually labeled with their level of dialectness. We provide a detailed analysis of AOC-ALDi and show that a model trained on it can effectively identify levels of dialectness on a range of other corpora (including dialects and genres not included in AOC-ALDi), providing a more nuanced picture than traditional DI systems. Through case studies, we illustrate how ALDi can reveal Arabic speakers' stylistic choices in different situations, a useful property for sociolinguistic analyses.

ALDi:
Quantifying the Arabic Level of Dialectness of Text
Amr Keleg, Sharon Goldwater, Walid Magdy Institute for Language, Cognition and Computation School of Informatics, University of Edinburgh akeleg sms ed. ac uk, {sqwater, wmagdy) } @ linf, ed. ac uk
Abstract
Level Dialectness
Egy
Levantine
MSA
جن الر أسددنا
جلل الر أسعدنا
Transcribed speech and user-generated text in Arabic typically contain mixture of Modern Standard Arabic (MSA), the standardized lan- guage taught in schools, and Dialectal Ara- bic (DA), used in daily communications. To handle this variation, previous work in Arabic NLP has focused on Dialect Identification (DI) on the sentence or the token level. However, DI treats task as binary, whereas we ar- gue that Arabic speakers perceive spectrum of dialectness, which we operationalize at the sentence level as the Arabic Level of Dialect- ness (ALDi), a continuous linguistic variable. We introduce the AOC-ALDi dataset (derived from the AOC dataset), containing 127,835 sen- tences (17% from news articles and 83% from user comments on those articles) which are manually labeled with their level of dialectness. We provide a detailed analysis of AOC-ALDi and show that a model trained on it can effec- tively identify levels of dialectness on a range of other corpora (including dialects and genres not included in AOC-ALDi), providing more nuanced picture than traditional DI systems. Through case studies, we illustrate how ALDi can reveal Arabic speakers' stylistic choices in different situations, useful property for soci- olinguistic analyses.
Low
أسدداا الراجل
أسعدنا لمة افز
Medium
بسطف اجل الر
بسطنا لمة الز
High
شوييصاا اجلل الر
تقتونا للم الز
Table 1: Example sentence meaning the man cheered us written with different levels of dialectness in two Arabic dialects. Words with DA features are underlined. The dialectal sentences use their preferred SVO word order, contrasted by VOS order for MSA. The low di- alectness example also shows lexical dialectal feature for word the man (MSA J (²ر the Egyptian word (الل إلرر differs from MSA in single character, while the equivalent Levantine word (يهق (از has a different origin. Both dialects allow different variants for the verb: one variant (اسسان)), used in both dialects, shares a root with the MSA variant, while the more dialectal variants (هووننن Egyptian and لففدواا in Levantine) do not.
(Habash, 2010) sometimess even being mutually unintelligible (Abu Farha and Magdy, 2022) and they do not have a standard orthography.
These differences between MSA and DA, and the fact that speakers commonly code-switch be- tween the two, are a major challenge for Arabic NLP systems. As result, many systems have been designed to perform Dialect Identification (DI), often on the sentence level (Zaidan and Callison- Burch, 2011; Elfardy and Diab, 2013; Salameh et al., 2018), but also on the token level as a way of detecting code-switching points (Solorio et al., 2014; Molina et al., 2016). Both formulations take a binary view of the problem (a sentence or token is either MSA or DA), and assume all the features of DA have the same impact on the perceived "dialect- ness" of sentence. We argue, however, that the level of dialectness of a sentence is a spectrum, illustrated in Table 1. Earlier initiatives recognized the presence such a spectrum (Habash et al., 2008; Zaidan and Callison Callison-Burch, 201 1), however,
1 Introduction
Arabic is spoken by more than 420 million people all over the world (Bergman and Diab, 2022), and exists in a state of Diglossia, in which two vari- ants of the language co-exist in Arabic-speaking communities (Ferguson, 1959). Modern Standard Arabic (MSA) is the standardized variant, which is taught in schools and used in formal communi- cations and as common language across all Arab countries. However, many local variants of Dialec- tal Arabic (DA) are used for daily communication mainly in speech and speech- text such as social media. They can diverge from MSA and each other in phonology, morphology, syntax, and semantics

---

the datasets that were developed are either skewed toward more standardized documents with limited code-switching or lack information about the dis- tribution and the quality of the levels of dialectness labels. Consequently, the Level of Dialectness has not yet been adopted as linguistic variable that is formally recognized in analyzing Arabic text, de- spite being potentially useful for NLP applications.
While Arabs can understand and read this standard language, spontaneously speaking in the standard language is not a natural task for most of them. Variants of DA are generally used in everyday com- munications, especially in spontaneous situations, and are widely used on social media platforms.
DA variants can be grouped on the level of re- gions (5 major variants: Nile Basin, Gulf, Levant, Maghreb, and Gulf of Aden), countries (more than 20 variants), or cities ( (100+ variants) (Baimukan et al., 2022). In text, MSA differs from DA in terms of morphemes, syntax, and orthography. These dif- ferences form cues of dialectness in code-switched text. In the orthography, distinctive DA terms are written in ways that match the pronunciation. Re- gional differences in the pronunciation of MSA terms are typically lost in writing due to the stan- dardized orthography, but in some cases, individ- uals use non-standard orthography that matches their regional pronunciations (c.g.: Man written as راجل instead of the standardized form رب , as in Table 1).
We argue that the level of dialectness is an impor- tant but overlooked aspect of Arabic text which is complementary to, and more nuanced than, dialect identification. To support this claim and promote further research in the area, we:
1. Define the Arabic Level of Dialectness (ALDi) as a continuous linguistic variable that quantifies the dialectness of sentence (or sentence-like unit) and can enrich the analysis of Arabic text.
2. Release AOC-ALDi a dataset of 127,835 Arabic comments with their ALDi labels. which is derived from the Arabic Online Com- mentary dataset (Zaidan and Callison-Burr Burch,, 2011). We provide the first detailed analy- sis of the level of dialecmess labels and form canonical splits for the AOC-ALDi dataset.
Arabic Dialect Identification Due to the rise of social media text, handling DA has become in- creasingly important for Arabic NLP systems. To date, researchers have focused on Dialect Identifi- cation (DI), which can be modeled either as a bi- nary MSA-DA classification lem with a prespecified set of DA variants (Al- thobaiti, 2020; Keleg and Magdy, 2023). Arabic DI has attracted considerable research attention, with multiple shared tasks (Zampieri et al. 2014; Bouamor et al. 2019; Abdul-Mageed et al. 2020, 202 lb, 2022) and datasets (Zaidan and Callison- Burch, 2011; Bouamor et al., 2014; Salama et al., 2014; Bouamor et al., 2018; Alsarsour et al., 2018; Zaghouani and Charfi, 2018: El-Haj. 2020; Abde- lali et al., 2021; Althobaiti, 2022).
3. Propose an effective method for estimating the ALDi of sentences, that can generalize to corpora of other genres and dialects 2.
multi-class prob-
4. Demonstrate via case studies that ALDi esti- mation of transcribed political speeches can highlight interesting insights that existing DI systems fail detect.
We hope that our work on the Level of Dialect- ness variable can motivate research in this direc- tion applied to other languages such as Swiss Ger- man where Standard variant co-exists with non- standardized ones.
2 Background and Related Work
Much of this work has been done at the sentence or document level, but there has also been work on token-level DI for code-switching. for example on Egyptian ArA tweets (Solorio et al., 2014; Molina al., 2016) and on Algerian Arabic (Adouane and Dobnik, 2017).
MSA and Dialectal Arabic Unlike English. where there is no single standard variant used in all English-doninant countries, Arabic speakers agree great extent having single standard- ized form of the language that they call Fus-ha ففسحي Arabs perceive both MSA and Classical Arabic (CA), the variant of Arabic that dates back to the 7th century, as Fus-ha (Parkinson, 1991).
Levels of Dialectness Both sentence-level and token-level DI methods fail distinguish between sentences having the same number of dialectal cues, yet different levels of dialectness. As per Table 1, each of the sentences بسطنا للة ازز and ادJ الز فققوياا has two lexical cues of dialectness, yet the
'The code and data files can be accessed through https: / //githtb... com/ AMR KELEG/ALD /AAD
'A live demo for AL.i estimation: https:// hugging co/ spaces/ae MERR-KLLEG/AD

---

latter sentence is perceived being more dialectal than the former. Only a very few works have con- sidered this distinction. One is Zaidan and Callison- Burch 2011), who collected sentence-lewe dialect- ness annotations in the Arabic Online Commentary data set. Although the dataset has been released, there has been no published description or analysis of these annotations that we know of, and (perhaps for this reason) no follow-up work using them Our work aims to remedy this.
Type
AIGhad
AlRiyadh
Youm7
Comment
94,236
156,345
80,349
Control
48,210
8,925
9,051
All
142,446
165,270
89,400
Table 2: Statistics of the AOC dataset, showing the num-
ber of annotations of each type from each newspaper source. Each sentence has independent annotations.
ing regional dialect). Therefore, believe the level of dialectness and formality are related yet not interchangeable.
An earlier project that annotated dialectness was Habash et al. (2008), who proposed a word-level an- notation scheme consisting of four levels: (1) Pure MSA, (2) MSA with non-standard orthography, (3) MSA with dialect morphology, and (4) Dialec- tal lexeme. Annotators also labeled full sentences according to their level of dialectness. Although the anter-annotator agreement was relatively good (less so for the sentence level), only a small corpus was annotated (19k words). Moreover, the corpus has sentences that are mostly in MSA with limited code-switching. A later work piloted simplified version of the scheme another corpus of 30k words (Elfardy and Diab, 2012). Both corpora not publicly released.
The Arabic Level of Dialectness (ALDi)
We define the Level of Dialectness of sentence the extent by which the sentence diverges from the standard language, which can be based on any of the cues described above. This definition is con- sistent with the crowd-sources annotation of the Arabic Online Commentary (AOC) dataset (Zaidan and Callison-Burch, 2011), where annotators la- beled user comments Arabic newspaper articles by their dialect and their level of dialectness. How- ever, the original and subsequent work only used the dialect labels, and the dialectness annotations have not previously been analyzed in detail.
Level of Dialectness and Formality Formality is a concept that has been studied, yet it does not generally have an agreed-upon definition (Hey- lighen and Dewaele, 1999; Lahiri, 2016; Pavlick and Tetreault, 2016; Rao and Tetreault, 2018). Hey- lighen and Dewaele (1999) define formality as the avoidance of ambiguity by minimizing the context- dependence, and the fuzziness of the used expres- sions. Later operationalizations recognize factors such as slang words and grammatical inaccuracies have on the people's s perception of formality (Mos- quera and Moreda, 2012; Peterson et al., 2011) as cited in (Pavlick and Tetreault, 2016).
3.1 Analyzing the AOC Dataset
The AOC dataset was created by scraping user comments on articles from three different newspa- pers, which are published in Egypt (Youm7 م اليو االا Jordan (AIGhad أال1), and Saudi Arabia (AlRiyadh- االريافي thus expecting the major- ity of comments of each source to be in Egyptian (EGY), Levantine (LEV), and Gulf (GLF) dialects respectively, Each comment is labeled for its level of dialectness (MSA, little, mixed, mostly dialectal, not Arabic). For comments labeled as Non-MSA, the annotators also chose the dialect in which the text is written: EGY, LEV, GLF, Maghrebi (MAG), Traqi (IRQ), General (GEN: used when the text is DA, but could belong to multiple dialects), Unfa- miliar, and Other.
Arabic speakers tend to use MSA in formal situ- ations, and their regional dialects in informal ones. However, an Arabic speaker can still use MSA and speak informally, or use their dialect and speak for- mally. The case studies described in $5 show how Arab presidents use sentences of different levels of dialectness in their political speeches. While these speeches would all be considered to be formal. us- ing different levels of dialectness might be to sound authoritative (using MSA) or seek sympathy (us-
Each row of the released AOC dataset consists of 12 different sentences representing a Human Intelligence Task (HIT) on Amazon m n'Turk, with annotations provided by the same human judge. A HIT has 10 comments in addition to 2 control sen- tences sampled from the articles' bodies, which are expected be mostly written in MSA. As part of each HIT, annotators provided some personal in- formation such as their place of residence, whether
"This contrasts with the authors' DI annotations for the same corpus, which were analyzed in Zaidan and Callison- Burch 2)014), and have been widely used in DI tasks.

---

Type
MSA
Little 36,930
Mixed
Most 76,284 (23.05%)
Not Arabic
Missing
Comment
189,020 (57.12%)
16%)
21,622 (6.53%)
5,421 (1. 646)
1,653 (0.5%)
Control
62,456 (94.36%)
1,060 (1.6%)
436 (0.66%)
754 (1.14%)
1,165 (1.76%)
315 (0.48%)
AII
251,476 (63. 33%)
37,990 (9.57%)
22,058 (5.55%)
77,038 (19.4%)
6,586 (1.66%)
1,968 (0.5%)
Table The distribution of AOC's Level of Dialectness annotations. Each sentence has 3 independent annotations. Control are sentences extracted from the article body, most likely MSA, to check the quality of the annotations.
number of annotations per source (Table 2) is more skewed. This likely due to the fact (noted by Zaidan and Callison- Burch 2014) that AlGhad con- tains the highest proportion of MSA annotations, followed by AlRiyadh and then Youm7. Figure I also shows that the distribution of dialectness lev- els is similar for the LEV, GLF, and EGY dialects, whereas the GEN dialect label has a higher propor- tion of little dialectness. This makes sense, since for sentences with few cues of dialectness, the level of dialectness would be low, and it would hard to assign these sentences to specific dialect.
No. of annotations
40000
39507
36916
35755
ALDI
27.3%
24.0%
Mixed
16.8%
14.15
211882
Mast
20000
Arabic
40.0%
Missiing
10000
55.9%
6.1.99
59..8%
3..99%
42.0%
2695
LEV
G41
EG
GEN
- ARA
MISSING
REST
Dialect label
The distribution of the annotations for the dialect and level of dialectness in AOC. Note that each comment has three different annotations. 251,476 MSA annotations are not shown in the Figure. The General dialect label is used when a sentence natural in multiple variants of DA. The REST bar represents the (Maghrebi, Iraqi, Unfamiliar, and Other) labels.
Figure 1:
3.2 From AOC to AOC-ALDi
In order to transform the AOC level of dialectness annotations into numeric ALDi scores, applied the following steps:
Step #1 HIT to annotation rows: We split each row (HIT) of the AOC dataset into 12 annotation rows, one for each sentence of the HIT, with the annotator's information shared across them.
No, of samples
67153
50000
25000
21265
10091
21326
Step #2 Grouping identical comments: Com- ments on the same article can sometimes be iden- tical. We decided to group identical comments on the same article together. Out of 129,873 grouped comments, only 1,377 comments have more than three annotations. We discarded 2,038 grouped comments for which at least 3/z of the dialectness level annotations either Missing Not Arabic, leaving a total of 1 127,835 comments with least three annotations each. The average length of these comments is 20 words.
10.0.111
[..11,0..4[
10.44,
[0.77.1[
Aggregated ALDi Score
Figure 2: AOC-ALDi's distribution of ALDi scores.
they are native Arabic speakers, and the Arabic dialect they understand the most. Table 2 shows the number of annotations collected for sentences from each source.
Table 3 shows the distribution of Level of Di- alectness annotations in AOC. As expected, the control sentences are nearly all (94%) annotated MSA. MSA is also the most common label for the scraped comments (57% of their annotations), followed by the mostly dialectal label (23%), little dialectal (1),, and mixed (6.5%).
We measured inter-annotator agreement on the level of dialectness annotations for the 124,257 comments which have 3 annotations that are not Not Arabic or Missing. The Fleiss' Kappa (x) is 0.44 (Fleiss, 1971), while Krippendorff's Alpha (interval method) (a) 0.63 (Krippendorff, 2004). Both metries are corrected for chance agreement and disagreement respectively. y considers the labels as categorical, while penalizes disagree- ments according the differences between their val-
Figure shows the distribution of dialectness la- bels split out by dialect (sentences labeled as MSA are not shown). We see that the proportions of different levels of dialectness for the LEV, GLF, and EGY dialects are similar, even though the total
"The main categories of these discarded comments are discussed in Appendix $B

---

AlGhad
AlRiyadh
Youm7
Split
Cmnt
Cutrl
Cmnt
Cntrl
Cmnt
Cntrl
Train (80 ())%)
24,039
12,613
41,479
2,335
20,041
2.379
Dev (10%) %)
3,107
1,513
4,567
275
2,475
323
Test (10%)
2,945
1,587
5,012
360
2,514
271
Table 4: The number of grouped comments in AOC-ALDi's splits. 127,835 comments of 20 words average, are distributed across all splits.
Comment
English translation (ours)
ALDi
siity مسiر J مسوو atه القون الفاي الي م الورز العسيد وراوو
Bravo the wonderful Minister, who proved is respon- sible, feeling the importance of his responsibility for the first time in long time in the history Egyptian education.
0,0, 3
العايي تاريع في ببيد زمن مند مرد لاوو aينم مستو ادميه
Ps 0.11
of المسس
اليين الهر فير س المدار بر تططر النمح الغفل بقس فيني
We start with the right task of developing schools and provid-
J.1 0.56
علياا
ing observers them
طوو ماب lumn بففسلل بنت ياا بامر 2 وور
Honestly, a serious mimister .... hope he stays this all the time
1,
1.00
Table Sample comments to the same article with their level dialectness labels (3 annotations for each comment with their mean the ALDi score). The labels are MSA (0), Little (t),, Mixed (3), Most (1). DA segments are underlined. Loanwords are double-underlind.
ues. Although these agreement levels are consid- ered only moderate, our experiments demonstrate that the corpus can nevertheless useful.
loanword. The rest of the sentence has MSA terms that will not sound natural if pronounced according
to the phonetic rules variant of DA. Unsurpris- ingly, two annotators considered the sentence to be in MSA, while the third might have perceived the presence of the loanword as a sign of dialcctness, thus marking the sentence little dialectal. The second example shows code-switching between MSA and Egyptian DA, but Egyptian still naturally pronounce the MSA portion abiding by the phonetic rules of Egyptian Arabic. This might be the reason why one of the annotators labeled the sentence as mostly dialectal (see Parkinson (1991), )), who observed the same relation between pronunci- ation and perceived levels of dialectness). For the third example. all the tokens except for the first one show dialectal features, which made it easy for the three annotators to classify it as mostly dialectal.
Step #3 - Label aggregation: Multiple human annotations for the level of dialectness were ag- gregated into single label. We transformed the ordinal labels (MSA, Little, Mixed, Mostly) into the numeric values (0, 3 3
2 1), then took the al- gebraic mean of these as the gold standard label, which has the range [0, 1).5 The distribution of the aggregated scores across four intervals is shown in Figure 2.
To build reliable splits of AOC, we made sure comments the same doc- ument are in the same split. For each source, we group sentences belonging to the same article to- gether, shuffle these groups, and then assign the first 80% of the comments to the training split, the following 10% to the development split, and the remaining 10% to the test split. This way, the dev and test sets evaluate whether a model general- izes to comments from articles not seen in training. The distribution of the sources across AOC-ALDi's splits is in Table 4.
Step #4- Splits creation:
4 The ALDi Estimation Task
Before describing case studies demonstrating pos- sible uses of automatic ALDi estimation, we first show that a model trained to predict ALDi is com- petitive with a DI system in discriminating between dialects (including dialects barely represented in AOC-ALDi), while providing more nuanced di- alectness scores. We then consider several specific features of Egyptian Arabic, and again show that the ALDi regression model is more sensitive to these than the baseline approaches.
Qualitative Analysis: Table 5 shows three exam- ple sentences from the AOC-ALDi dataset with their corresponding annotations where all annota- tors labeled the dialect as either MSA, EGY, or GEN. The first sentence begins with an English
'AOC-ALDi also includes the original sepurate labels.

---

4.1 Models
Model
RMSH(L)
Cntrl
Cmnt
All
The main model we to predict ALDi BERT- based regression model. Using the training split of AOC-ALDi, we fine-tune regression head on top of MarBERT, an Arabic BERT model (Abdul- Mageed et al., 2021a), and clip the output to the range [0, 1]. To measure the consistency of the model's performance, we repeat the fine-tuning pro- cess three times using 30, 42, and 50 as the random seeds, and report averaged evaluation scores for the model (similarly for Baseline #3). We compare this model to three baselines, which use existing Arabic resources and are not trained on AOC-ALDi. Baseline #1 Proportion of tokens not found in an MSA lexicon: The presence of dialectal lexical terms is one of the main signals that humans use to determine dialectal text. Sajjad et al. (2020) built MSA lexicon from multiple MSA corpora. They then computed the percentage of tokens within sentence not found in the MSA lexicon as a proxy for sentence-level dialectness. We replicate this method using the tokens occurring more than once in the Arabic version of the United Nations Pro- ceedings corpus (Ziemski et al., 2016) as the source for the MSA lexicon.
N=2,127
N=10,644
N=12,771
MSA Lexicon
0.13
0.36
0.34
Sentence DI
0.23
0.53
0.49
Token DI
0.11*
0.33
0.30*
Sentence ALDi
0.07
0.19
0.18*
Table 6: Models' RMSE AOC-ALDi's test split.
*; Average values across three fine-tuned models with different random seeds. The corresponding standard deviations are 0.015 or less.
test split as the gold standard, we measure how the models' ALDi predictions deviate from the gold standard ones using Root Mean-Square Er- ror (RMSE). As expected since it is the only model trained on AOC-ALDi, the Sentence ALDi model achieves the least RMSE of 08 on the AOC-ALDi test split, indicated in Table 6. The two other models that produce continuous scores at the sentence level, MSA Lexicon and Token DI, achieve similar RMSE, and are both better than the binary Sentence DI model despite more limited exposure to the dialects in this corpus (recall that Token DI has only been trained on EGY and MSA, and MSA Lexicon has no explicit DA training). All models perform worse the comments than the controls.
Baseline #2- Sentence-Level DI: We use an off- the-shelf DI model implemented in (Obeid al., 2020) based on (Salameh et al., 2018). The model is based on Naive Bayes, trained on the MADAR corpus (Bouamor et al., 2018), and uses character and word n-grams to classify sentence into 6 variants of DA in addition to MSA. A sentence is assigned ALDi score of 0 if it is classified as MSA and score of otherwise.
Disentangling Parallel MSA/DA Sentences For model estimating ALDi, minimal requirement is to assign higher score to a DA sentence than that assigned its corresponding MSA translation. We utilize two parallel corpora of different gen- res and dialects to test this requirement. First, we use a parallel corpus of 8219 verses (sentences) from the Bible, provided by Sujjad et al. (2020), which includes versions in MSA, Tunisian, and Mo- roccan Arabic. We also use DIAL2MSA, which is a dataset of dialectal Arabic tweets with paral- lel MSA translations (Mubarak, 2018). Five MSA translations were crowd-sourced for 12,000 tweets having distinctive lexical features of Egyptian and Maghrebi Arabic. Each translation was then val- idated by 3 judges. For our analysis, we discard samples having a non- non-perfect validation confidence score, and ones that still have a distinctive dialectal lexical term in their MSA translations.
Baseline #3 - Token-level DI: Molina et al. (2016) created a token-level DI dataset (MSA-EGY to- ken DI), in which tokens of tweets were manually tagged as MSA, EGY, MameddEnntty ambiguous, mixed, other. We use this dataset to fine-tune layer on top of MarBERT to tag tokens of sen- tence. The tag of the first subword for each to- ken is adapted as the tag for the whole token as done in (Devlin et al., 2019). We use token-level tags to compute the Code-Mixing Index (CMI; Das and Gambäck 2014) as proxy for ALDi: NEGY -
CMI = NEGY tokens of the tokens are tagged as MSA or EGY).
(set to 0 if none
MSA tokens
The distribution of the ALDi scores in Figure 3 reveals that MSA Lexicon does not discriminate strongly between MSA and DA, while Token DI mostly assigns scores of 0 or 1 (acting like Sen- tence DI), despite the possibility to do otherwise.
4.2 Evaluation
Intrinsic AOC-ALDi evaluation Treating the aggregated human-are assiigned scores of AOC-ALDi's

---

and feminine) and word orders (SVO and VSO). While MSA allows for both word orders, it favors VSO (EI-Yasioe 1985), while Egyptian Arabic fa- vors SVO (Gamal- 1968 as cited in Holes, 2013; Zaidan and Callison-Burch, 2014). Ta- blc 7, we display the ALDi scores assigned by the different models to the contrastive pairs.
MSA Lexicon
1
Sentence
1
0
Token
The MSA Lexicon model considers all dialectal features to have the same impact in assigning a non-zero ALDi score (i.e., RS 0.33 or 2 R 0.5) to the DA sentences. As implied by our previous experiment, the Token DI model acts as sentence- level DI model, tagging all the tokens as dialectal if only one token shows distinctive dialectal feature. This behavior might be artifact of the model's s fine-tuning dataset, where annotators were asked to use the surrounding context determine an am- biguous token's 's language (EGY or MSA).
1
0
Sentence ALDi
1
BIBLE BIBLE BIBLE (MSA)TUN)
DM: DMr (EGY)
DMW DM,
(MGR)
The distribution of the ALDi scores as- signed by the four models to sentences the Bible and DIAL2MSA corpora. Each column (across the four plots) represents the same set sentences as scored by the four different models, and the columns are grouped by corpus to compare the different dialectal versions of that corpus. For each plot, the orange line shows the median score, the box represents the interquartile range (IQR) [Q1, Q3) of the scores, the whiskers represent =1.5 A(IQR) beyond QI and Q3, and the dots repre- outliers beyond this. Notel: A(IQR) Q3 Q1. Note2: The boxplots for the Token DI and Sentence ALDi models are not significantly different across the multiple fine-tuning runs of different random seeds.
Figure 3:
Conversely, the Sentence ALDi model provides a more nuanced distinction between the different fea- tures. The negation form (F4, F5) used in Egyptian Arabic seems to cause the model to categorically consider the sentence highly dialectal. Less salient features such as the (FI) present progressive proclitic increase the ALDi level of the sentence, but to a lesser extent than the negation feature. We also see that the model assigns higher ALDi scores to SVO sentences than VSO, suggesting that the model may have learned the common word order in Egyptian Arabic. Finally, tences tend to get higher scores compared to their mascullneemmrreed counterparts, which may be in- dicative of a gender bias in the training data and resulting model-if feminine marking is less com- mon, it may also be seen as less standard language and interpreted as non-MSA.
marked sen-
The Sentence ALDi model provides more nuanced scores while also showing strong discrimination be- tween MSA and DA, even for DA variants that are barely present in AOC-ALDi (TUN, MOR, MGR; note that Token DI also has not seen these). also yields slightly lower scores for the DA versions of the Bible than for the DA tweets, indicating that the informal genre of tweets may be an indicator of stronger dialectness levels.
Case Studies (ALDi in Practice)
The same speaker can adapt different styles accord- ing to various social and linguistic factors (Kiesling, 201 I). The ALDi of speech is one example an intraspeaker variation in Arabic. In this section, we provide two case studies analyzing the transcribed speeches of three different Arab presidents. We highlight how quantitatively estimating the ALDi can help in revealing different speaking styles.
4.3 Analysis Minimal Contrastive Pairs
Inspired by Demszky et al. (2021)'s corpus of min- imal contrastive pairs for 18 distinctive features of Indian English, we build contrastive pairs of MSA and Egyptian Arabic variants single sen- tence. We investigate features Egyptian Ara- bic that were previously recognized by Darwish et al. (2014). For each sentence, we generate ver- sions with different gender markings (masculine
5.1 Presidential Speeches in the Arab Spring
Lahlali (2011) qualitatively analyzed the usage of MSA and DA (Tunisian Arabic and Egyptian Arabic) in the last three speeches of Ben-Ali and
Fuurter discussion, including /)' diserimination scores for all models, can be found in Appendix C.

---

LSA.EX.
SEN,
TOK. DI
ALD
Fratury tome
M
RGY
Word onler
MSA
EGY
MSA
BOY
MSA
BGY
MSA
BGY
F1] Present progreedre
المديية الي ت المعيعة تعر الب
المقيحة البيد J سترر
VS0
0.0
0.53
0.0
00/1.
1.0
1.0
01100122
0.55//555
The gil is uying that
المتيةة ن بشور اليي
RVO
100
1.0
0.370025
083/066
Luture Morpheme
Rapasdi الينك سللل
المقرية البنت J متدر
V30
0.0
0.0
3.0
0,20007
0.76.
will say
المقرمة سنار اووا السقرمة طوات
الحجرمة متتر
SVD
10700
0.11
0.02
/179
13) Passhe formation
السقيقة أتمادد
V0
0.0
0.5
0.0
0.0
0.0
1.0
0.05
0.36
Ra: The mnth was and
اونن السقونة
التااا المنينة
QV
0.
0.5
0.0
1.0
0.0
1.0
0.11
0.36
F4) Negation
الدديعة البنت ححر الدقيمة البند لنر مستام
V30
1.0
0.17
0.087011
0.55
saying wuth
R(caري) تعو المجدرة منتا
5VD
1.0
0.22
0.00 71.13
E5) Negataa imperative
الحليية لي الر
لنرر مالدر
VSO
0000.33
1.0
3.0
0.070.13
0.34/091
Ea: Do snt soy fae tuih
Table 7: The ALDi scores assigned to contrastive MSA and Egyptian Arabic sentences. Only the feminine-marked version of the sentence is shown, and tokens with dialectal features are underlined. A single score is reported if model assigns the same score the masculine and feminine versions of sentence, otherwise the scores for masculine/feminnne are shown. We tested VSO (favored in MSA) and SVO (favored in EGY) word orders.
Note: Scores E [0, 0.11] are encoded in green, while ones E j0. I] have a shade purple.
*: The scores for these models averaged across three fine-tuned models with different random seeds.
Mubarak, the former Tunisian and Egyptian presi- dents during the period of the Tunisian and Egyp- tian revolutions. Mubarak consistently used MSA for his speeches to showcase authority and power. Ben-Ali used MSA for his first two speeches. For his last speech, he explicitly said: "2iJ تكلوتكم فسبات و فسيينن التو كك" "I talk to you in the language of all the Tunisians", apparently using his choice of dialect as a way to identify himself with a particular group (cf. Shoemark et al. 2017; McNeil 2022).
from a almanassa com. The transcripts are not limited to the edited presidential speech, but also include greetings, introductory comments, inter- ventions by the audience, and signs of disfluency or hesitation. The site's editors segmented each speech into coherent sentences, embedded in <p> HTML tags, that we adapt as units of analysis. While most of these speeches are conducted in MSA, multiple cases of code-switching between MSA and Egyptian Arabic occur. For example, in Figures 4e and 4f, El-Sisi used MSA when reading the edited speech, and Egyptian Arabic with high ALDi scores when spontaneously addressing the audience before or after the edited speech.
We quantitatively replicate the analysis by visu- alizing the ALDi scores of the transcribed speeches. We scraped the speeches from online websites? and used the HTML line breaks <br> to segment them into sentences. For each sentence, we predict the ALDi score with our model and also use the Sen- tence DI model to classify it as DA or MSA.
Interestingly, Figure 4g shows three different ALDi levels as part of the same speech. El-Sisi used MSA for reading the edited speech directed to the press, discussing issues such Egyptian- German diplomatic relations, climate change. and economic hardships. He then reacted sponta- neously to two questions from the press. He at- tempted to answer the first question, related to gas prices, in MSA but the sentences show code- switching between MSA and Egyptian Arabic, in- dicated by intermediate ALDi scores (though the DI system does not identify these). For the sec- ond question about human rights in Egypt, El-Sisi uses sentences that are more dialectal and less for- mal, inviting the journalist to visit Egypt in order to make a fair assessment of the situation. This is indicated by even higher ALDi scores. Samples from each segment are listed in Appendix D.
Figure 4a shows that our model correctly finds nearly 0 ALDi scores throughout Ben-Ali's speech on the 1hth of January, while the DI model makes couple of errors (and similarly for Mubarak's speeches, shown in Figures 4c, 4d). Both mod- els identify the shift to DA in the second speech (Figure 4b), with more sentences identified as DA by the DI model, and many with moderate ALDi scores. Given the nature of the speech, Ben-Ali still used formal terms while speaking in Tunisian Ara- bic which is likely the reason for the intermediate ALDi scores.
5.2 El-Sisi's Speeches
Next, we studied the ALDi scores for 659 speeches of the current Egyptian president El-Sisi, scraped
This speech is a clear example of how an Arabic speaker can adapt different levels of dialectness in their speech and indicates the ability of ALDi to reveal such differences.
www. www.bbabbttttccm. and egypt-bl beypt ew.bl ogspot.
com

---

a) Ben-Ali 10/1/2011
b) Ben-Ali 13/1/2011
ALD.
DA
ALDI
MSA
0
10
20
30
40
10
20
Santence Index In Speech
Sentence Index in Speech
c) Mubarak 1/2/2011
d) Mubarak 10/2/2011
ALD
20
20
30
Sentence Index in Speech
Sentence Index in Speech
e) El-Sisi 7/12/2017
f) EI-Sisi 10/10/2021
g) El-Sisi 22/7/2022
Q&A Human Rights
06A
ALDI
Gas
Presidential Speech
10
10
20
Santence Index Speech
Sentence Index Speech
Sentance Index Speach
Figure 4: The ALDi scores assigned sentences of transcribed political speeches. Subfigures and h) represent two speeches of the former Tunisian president Ben-Ali during the Tunisian Revolution. Subfigures c) and d) represent two speeches the former Egyptian president Mubarak during the Egyptian Revolution. Subfigures e). f), and g) are speeches of the current Egyptian president El-Sisi. The MSA/DA labels were generated by the Sentence DI model.
6 Conclusion
periments show robustness across multiple genres of text, will be interesting prepare dataset (even just for intrinsic testing) that comes from other sources, such as social media. Reannotating existing DI datasets with ALDi might be a first-to- do option.
We presented ALDi, linguistic variable that quan- tifies the level of dialectness of an Arabic sentence. We release AOC-ALDi, a dataset of Arabic com-- ments annotated with their ALDi scores. A BERT- based regression model fine-tuned on AOC-ALDi showed superior performance compared to existing baselines that are based on lexicons and DI mod- els. Our analysis shows that the model generalizes to various Arabic dialects. In addition, the model provides nuanced distinction of dialectal features, which token and sentence DI models can not per- form. Lastly, we presented multiple case studies demonstrating the effectiveness of ALDi in reveal- ing new insights in Arabic text. For future work, we aim to explore the possible applications of ALDi for text analysis, especially for sociolinguistics and computational social science studies. Moreover, we aim to apply the level of dialectness work to other languages that have the same phenomena of Arabic, such as
Also, the gold-standard ALDi scores in our AOC- ALDi dataset are based on normalizing the level of dialectness annotations of the AOC dataset, which might be sub-optimal. Labeling a dataset directly with continuous ALDi scores might provide more accurate labels (still might be more challenging for annotators).
While our experiments cover diverse dialects of Arabic, the generalizability of ALDi for more dialects of Arabic more dialects needs to be tested.
Finally, found preliminary evidence of possi- ble gender bias in our dataset/model. While we did not explore this issue depth here, will impor- tant to consider impact and possible mitigation strategies in future work.
Limitations
Acknowledgments
Our AOC-ALDi dataset is based on the AOC dataset that comes mainly from news comments, which might be specific genre. Although our ex-
This work was supported by the UKRT Centre for Doctoral Training in Natural Language Processing, funded by the UKRI (grant EP/S022481//) and the

---

University of Edinburgh. School of Informaties. We thank Nizar Habash for his valuable input regarding his previous efforts related to the idea of the level of dialectness.
Israa Alsarsour, Esraa Mohamed, Reem Suwaileh, and Tamer Elsayed. 2018. DART: A large dataset of di- alectal Arabic tweets. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan. Eu- ropean Language Resources Association (ELRA).
Maha J. Althobaiti. 2020.
Automatic arabic dialect identification systems for written texts: A survey.
References
Ahmed Abdelali, Hamdy Mubarak, Younes Samih, Sabit Hassan, and Kareem Darwish. 2021. QADI: Arabic dialect identification in the wild. In Proceed- ings of the Sixth Arabic Natural Language Process- ing Workshop, pages 1-10, Kyiv, Ukraine (Virtual). Association for Computational Linguistics.
Maha J. Althobaiti. 2022. Creation of annotated country-level dialectal arabic resources: An unsu- pervised approach. Natural Language Engineering, 28(5):607-648
Nurpeiss Baimukan, Houda Bouamor, and Nizar Habash. 2022. Hierarchical aggregation of dialectal data for Arabic dialect identification. In Proceedings of the Thirteenth Language Resources and Evaluation Con- ference, pages 4586-4596, Marseille, France. Euro- pean Language Resources Association.
Muhammad Abaul-Mageed, AbdelRahim Elmadany, and EI Moatez Billah Nagoudi. 2021a. ARBERT & MARBERT: Deep bidirectional transformers for Ara- bic. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the
1 11th International Joint Conference on Natural Lan- guage Processing (Volume 1: Long Papers), pages 7088-7105, Online. Association for Computational Linguistics.
A. Bergman and Mona Diab. 2022. Towards respon- sible natural language annotation for the varieties of Arabic. In Findings of the Association for Com- putational Linguisties: ACL 2022, pages 364-371, Dublin, Ireland. Association for Computational Lin- guistics.
Muhammad Abdul-Mageed, Chiyu Zhang. Houda Bouamor, and Nizar Habash. 2020. NADI 2020: The first nuanced Arabic dialect identification shared task. In Proceedings of the Fifth Arabic Natu- ral Language Processing Workshop, pages 97-1 10, Barcelona, Spain (Online). Association for Computa- tional Linguisties.
Houda Bouamor, Nizar Habash, and Kemal Oflazer. 2014. A multidialectal parallel corpus of Arabic. In Proceedings of the Ninth International Conference on Language Resources and Evaluation 14), pages 1240-1245, Reykjavik, Iceland. European Lan- guage Resources Association (ELRA).
Muhammad Abdul- Magul-Mageed, Chiyu Zhang, AbdelRahim Elmadany, Houda Bouamor, and Nizar Habash. 1b. NADI 2021: The second nuanced Arabic dialect identification shared task. In Proceedings of the Sixth Arabic Natural Language Processing Workshop, pages 244-259, Kyiv, Ukraine (Virtual). Association for Computational Linguistics.
Houda Bouamor, Nizar Habash, Mohammad Salameh, Wajdi Zaghouani, Owen Rambow, Dana Abdul- rahim, Ossama Obeid, Salam Khalifa, Fadhl Eryani, Alexander Erdmann, and Kemal Oflazer. 2018. The MADAR Arabic dialect corpus and lexicon. In Pro- ceedings of the Eleventh International Conference Language Resources and Evaluation (LREC 2018). Miyazaki, Japan. European Language Resources As- sociation (ELRA).
Muhammad Abdul-Mageed Chiyu Zhang, AbdelRahim Elmadany, Houda Bouamor, and Nizar Habash. 2022. NADI 2022: The third nuanced Arabic dialect identi- fication shared task. In Proceedings of the The Sev- enth Arabic Natural Language Processing Workshop (WANLP), pages 85-97, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computational Linguistics.
Houda Bouamor, Sabit Hassan, and Nizar Habash. 2019. The MADAR shared task on Arabic fine- fine-grained di- alect identification. In Proceedings of the Fourth Ara- bic Natural Language Processing Workshop, pages 199- Florence, Italy. Association for Computa- tional Linguistics.
Kareem Darwish, Hassan Sajjad, and Hamdy Mubarak. 2014. Verifiably effective Arabic dialect identifi- cation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1465-1468, Doha, Qatar. Associa- tion for Computational Linguistics.
Ibrahim Abu Farha and Walid Magdy. 2022. The effect of Arabic dialect familiarity data annotation. In Proceedings of the The Seventh Arabic Natural Lan- guage Processing Workshop (WANLP), pages 399- 408. Abu Dhabi, United Arab Emirates (Hybrid). As- sociation for Computational Linguistics.
Amitava Das and Björ Gambäck. 2014. Identifying languages at the word level in code-mixed Indian social media text. In Proceedings of the 11th Interna- tional Conference on Natural Language Processing, pages 378-387, Goa, India. NLP Association of In- dia.
Wafa Adouane and Simon Dobnik. 2017. Identification of languages in Algerian Arabic multilingual docu- ments. In Proceedings of the Third Arabic Natural Language Processing Workshop. pages 1-8, Valnciaa, Spain. Association for Computational Linguistics.

---

Dorottya Demszky, Devyani Sharma, Jonathan Clark, Vinodkumar Prabhakaran, and Jacob Eisenstein. 2021. Learning to recognize dialect features. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Compula- tional Linguistics: Human Language Technologies, pages 23 15-2338, Online. Association for Computa- tional Linguistics.
Francis Heylighen and Jean-Mare Dewaele. 1999. For- mality of language: definition, measurement and be- havioral determinants. Interner Bericht, Center "Leo Apostel", Vrije Universiteit Brüssel, 4.
Clive Holes. 2013. Word order and textual function in gulf arabic. In Information structure in spoken Arabic, pages 79-92 92. Routledge.
Amr Keleg and Walid Magdy, 2023. Arabie dialect iden- tification under scrutiny: Limitations of single-label classification. In Proceedings of the first Conference on Arabie Natural Language Processing, ArabicNLP. Singapore (Hybrid). Association for Computational Linguisties.
Jacob Devlin, Ming-Wei Chang. Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Tech- nologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Scott F. Kiesling. 2011. Linguistic Variation and Change, volume 1. Edinburgh University Press.
Klaus Krippendorff. 2004, Content Analysis, an Intro- duction to Its Methodology, second edition. Thou- sand Oaks, CA: Sage Publications.
Mahmoud El-Haj. 2020. Habibi multi dialect multi national Arabic song lyries corpus. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 1318-1326, Marseille, France. Eu- ropean Language Resources Association.
Shibamouli Lahiri. 2016. Squinky! corpus of sentence-level formality, informativeness, and im- plicature.
Mohammed K. El- Yasin, 1985. Basic word order in classical arabic and jordanian arabic. Lingua, 65:107- 122.
M Lahlali. 2011. The arab spring and the discourse of desperation: shifting from an authoritarian discourse to a "demoeratic one". The Journ. of Arah Media and Society- Cairo: American Univ. in Cairo, (13).
Heba Elfardy and Mona Diab. 2012. Simplified guide- lines for the creation of large scale dialectal Arabic annotations. In Proceedings of the Eighth Interna- tional Conference on Language Resources and Eval- uation (LREC" 12), pages 371-378, Istanbul, Turkey. European Language Resourees Association (ELRA).
Karen McNeil. 2022. 'we don't speak the same lan- guage: language gaage''anggugge choice and identity on a tunisian internet forum. International Journal of the Sociol- ogy of Language, 2022(278):51
Giovanni Molina, Fahad AlGhamdi, Mahmoud Ghoneim, Abdelati Hawwari, Nicolas Rey- Villamizar, Mona Diab, and Thamar Solorio. 2016. Overview for the second shared task on language identification in code-switched data. In Proceed- ings of the Second Workshop on Computational Approaches to Code Switching, pages 40-49, Austin, Texas. Association for Computational Linguisties.
Heba Elfardy and Mona Diab. 2013. Sentence level dialect identification Arabic. In Proceedings of the 51st Anmual Meeting of the Association for Compu- tational Linguistics ( (Volume 2: Short Papers), pages 456-461, Sofia, Bulgaria. Association for Computa- tional Linguistics.
Charles A Ferguson. 1959, Diglossia. word, 15(2):325- 340.
Alejandro Mosquera and Paloma Moreda. 2012. A qualitative analysis of informality levels in web 2.0 texts: The facebook case study. In Proceedings of the LREC workshop: @ NIP can u tagl user generated content, pages 23-29.
Joseph L Fleiss. 1971. Measuring nominal scale agree- ment among many raters. Psychological bulletin, 76(5):378.
Hamdy Mubarak. 2018. Dial2msa: A tweets corpus for converting dialectal arabic to modern standard arabic. In Proceedings of the Eleventh International Confer- ence Language Resources and Evaluation (LREC 2018), Paris, France. European Language Resources Association (ELRA).
Saad Gamal-Fldin. 1968. syntactic study of Egyptian colloquial Arabic / By Saad M. Gamal-E Eldin. Janua linguarum. Series practica: 34. Mouton, The Hague.
Nizar Habash, Owen Rambow, Mona Diab, and Reem Kanjawi-Faraj 2008. Guidelines for annotation of arabic dialectness. Proceedings of the LREC Work- shop on HLT & NLP within the Arabic world, pages 49- 53.
Ossama Obeid, Nasser Zalmout. Salam Khalifa, Dima Taji, Mai Oudah, Bashar Alhafni, Go Inouc, Fadhl Eryani, Alexander Erdmann, and Nizar Habash. 2020. CAMeL tools: An open source python toolkit for Arabic natural language processing. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 7022-7032, Marseille, France. Eu- ropean Language Resources Association.
Nizar Y. Habash. 2010. Introduction to Arabic natural language processing, I edition, volume 3 of Synthesis Lectures on Human Language Technologies. Morgan and Claypool Publishers.

---

Dilworth Parkinson. 1991. Searching for modern fusha: lefe-life formal arabic. al- Arabiiyaa 24:31- 64.
Code Switching, pages 62-72. Doha, Qatar, Associa- tion for Computational Linguistics.
Mohammad Ali Yaghan. 2008, "arabizi": A contempo- rary style of arabic slang. Design Issues, 24(2):39- 52.
Ellie Pavlick and Joel Tetreault. 2016. An empiri- cal analysis of formality in online communication. Transactions of the Association for Computational Linguistics, 4:61-74.
Wajdi Zaghouani and Anis Charfi. 2018. Arap-tweet: A large multi-dialect Twitter corpus for gender, age and language variety identification. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan. European Language Resources Association (ELRA).
Kelly Peterson, Matt Hohensee, and Fei Xia. 2011 I. Email formality in the workplace: A case study on the Enron corpus. In Proceedings of the Workshop on Language in Social Media (LSM 2011), pages 86- 95, Portland, Oregon. Association for Computational Linguistics.
Omar F. Zaidan and Chris Callison-Burch 2011. The Arabie online commentary dataset: an annotated dataset of informal Arabic with high dialectal con- tent. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Hu- man Language Technologies, pages 37-41, Portland, Oregon, USA. Association for Computational Lin- guistics.
Sudha Rao and Joel Tetreault. 2018. Dear sir madam, may I introduce the GYAFC dataset: Corpus, bench- marks and metries for formality style transfer. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 129-140, New Or- leans, Louisiana, Association for Computational Lin- guistics.
Omar Zaidan and Chris Callison-Buree 2014. Ara- bic dialect identification. Computational Linguistics. 40( 1)117-200.
Hassan Sajjad, Ahmed Abdelali, Nadir Durrani, and Fahim Dalvi. 2020. AraBench: Benchmarking dialectal Arabic-English machine translation. In Proceedings of the 28th International Conference on Computational Linguistics, pages 5107, Barcelona, Spain (Online). International Committee on Computational Linguistics.
Marcos Zampieri, Liling Tan, Nikola Ljubešić, and Jörg Tiedemann. 2014. A report on the DSL shared task 2014. In Proceedings of the First Workshop on Ap- plying NLP Tooly Similar Languggss Varieties and Dialects, pages 58-67, Dublin, Ireland. Associa- tion for Computational Linguistics and Dublin City University.
Ahmed Salama, Houda Bouamor, Behrang Mohit, and Kemal Oflazer. 2014, YouDACC: the Youtube di- alectal Arabic comment corpus. In Proceedings of the Ninth International Conference on Language Resources and Evaluation (L.REC' 14), pages 1246- 1251, Reykjavik, Iceland. European Language Re- sources Association (ELRA).
Michal Ziemski, Marcin Junczys-Dowmunt Dowmunt, and Bruno Pouliquen. 2016. The United Nations parallel cor- pus v1.0. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC"16), pages 3530-3534, Portorož, Slovenia. European Language Resources Association (ELRA).
Mohammad Salameh, Houda Bouamor. and Nizar Habash. 2018. Fine-grained Arabic dialect identifica- tion. In Proceedings of the 27th International Con- ference on Computational Linguisties. pages 1332- 1344, Santa Fe, New Mexico, USA. Association for Computational Linguistics.
A AOC Annotation Interface
Zaidan and Callison-Burch (2011) used Amazon Mechanical Turk to annotate Arabic comments they scraped from three different newspapers. They provided the annotators with minimal guidelines for determining the dialect and level of dialectness of the comments. A screenshot of their annotation interface is shown in Figure A1.8
Philippa Shoemark, Debnil Sur, Luke Shrimpton, lain Murray, and Sharon Goldwater. 2017. Aye or naw, whit dae ye hink? Scottish independence and lin- guistic identity social media. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume
While the guidelines are minimal, we think that the Arabic and English translations of the labels might have impacted the annotator's understanding of the labeling process. For instance, the annotation interface has the Not Arabic label translated to (فق) 3 ووو Ii & أكرر in Arabic, which actually means
1. Long Papers. pages 1239-1248, Valencia, Spain. Association for Computational Linguistics.
Thamar Solorio, Elizabeth Blair, Suraj Mahar- jan, Steven Bethard, Mona Diab, Mahmoud Ghoneim, Abdelati Hawwari, Fahad AlGhamdi, Julia Hirschberg, Alison Chang, and Pascale Fung. 2014 14, Overview for the first shared task on language iden- tification code-switched data. In Proceedings of the First Workshop on Computational Approaches to
"The annotation site can be accessed through / www. cs. . juu.jju..e... edu a rcch ve/RcTMM 2011/ html /di/ a ect, classi fication.at.

---

Help Classify Arabic into Dialects!
aud spotion indemaadd
dakes Ao Agiall claal)) dxtegaah Parto
Belos will seroul Anale sereesess For resever
man dated (ipla) pertenne,
2. Thllla adia wablic dullet wrter inand.
This hilovingg may equiaa: the dialece:
(لااي)) "ypla? only Syrias,
Magfcotll (uoue dess NOT mas "Mirooua" only. Ialakka Varaccan has AUSIC Mggi lin Mighasi, Taaiiiii Maghotki and aad Libyaa Maglooki That's way all tare ooe per map.
Spelling (ult
syling dasiryy
qpeling
This is simppo tol and oou enseers will telp moooe reseach the Anable langeagn pleaee property planee hanc
Cod, plasee wweer those genalaon dost dillion
der' kow auvver gerstena evey canagh)
Aabii
Hewe
Aralil
you
Jysa
Whic wablo dialen
Witch Diabeet! diglt Apgil
Draket Lond
ApddAl Rogil hiii
Seatince Magil
lit Gol
Oeneral plawtad ar
Unitly Bull
Moedd (ieladl hald
95 [4]
Commnd Blln Aad
Figure A1: A screenshot of the annotation interface the AOC dataset (Zaidan and Callison- Callison-Burch, 2011).
Reason for Discarding
Sentence
Source
Level of Dialectness
$9999
Cmnt (Y7)
Arabic (x13),
Symbols
Missing (x2)
Cmnt (Ri)
Arabic (x3)
تليس üSoaaglavvs to protect the bahy from infection
Cmnt (Ri)
Arabic (x2),
English
MSA (x1)
agree with you that racism exists in the United States; also know it exists in Arab countries as well. Just remember that America elected black president with 360 electoral college votes. In terms of numbers, that means sweeping majority. Lets learn to be better than the Americans by developing our democratic systems for change...ece
Cmnt (Gh)
Arabic (x3)
very nice...
Cmnt (Ri)
Arabic (x3)
ya zamalek ya 7arameyaaaa
Cmnt (Y7)
Arabic (x2),
Arabizi
Most (xl)
ma howeh el blogs m3abbiyeh el denya ? yašni law doctor el jam3a bedo yet3ab shway yekteb articles, ma kan Jemel blog men zaman.
Cmnt (Gh)
Arabic (x2),
Most (x1)
http: /tte///eleeeeeeeee wwweemmmslllile imm ace st forum, htm attamddallt
Cmnt (Y7)
Arabie (x3)
URLs and Emails
Cutrl (Gh)
Arabic (x3)
5000 &#8730:DONE
Cmnt (Y7)
Arabic (x3)
Presence of HTML
href= Tttpe///ooopiioooappppoooooooooo ف أهر <</a>>
Cutrl (Y7)
Arabic (x3)
سيتممر من في بنا ق
Cmnt (Y7)
Arabic (x2)
Most (x1)
Table Al: Examples of the discarded AOC comments of majority labels set to Not Arabic or missing.
Note: Cmnt stands for comment, Cntrl stands for control sentence, Y7: Youm7, Ri: AlRiyadh, Gh: AlGhad.

---

(Another language, or symbols). We believe that Another language or symbols is not equivalent to Not Arabic, which might make annotators interpret the guidelines differently.
B Discarded Samples from AOC-ALDi
As mentioned in $3.2, discarded 2,038 com- ments that have the majority of their ALDi annota- tions either set to Not Arabic or are missing. Five different categorics such comments were iden- tified as per Table Al. These categories include sentences that have only punctuation marks, are written in English or Arabizi (Romanized Arabic (Yaghan, 2008)), are just links to sites emails, or have HTML encoded characters or formatting tags.
C Discrimination scores
For the experiments in $4.2, we computed D', a measure of discrimination, for all models on each pair of parallel corpora. Results are shown in Ta- ble C2. On the DIAL2MSA corpora, which likely more similar in style to AOC-ALDi, model performs about well Token DI, the other BERT-based model (which, like ours, has not seen MGR training), while also providing a wider range of scores (as shown $4.2). Token DI does somewhat better than our model on the Bible corpora, but again by making nearly binary judgments for each sentence.
D Edited and Spontaneous Speech
As depicted in Figure 4g, El-Sisi's speech on the 22nd of July 2022 can be split into three segments: the edited presidential speech, and two spontaneous responses from the president to questions from the succeeding press conference.
We sampled a sentence from each segment as shown in Table D3 to demonstrate the three dif- ferent levels of dialectness that categorize each segment.

---

Bible
DIAL2MSA
Model
MSA / TUN
MSA/MOR
MSA/ /EGY
MSA / /GGR
MSA Lexicon
1.28
1.55
1,48
1.73
Sentence DI
2.65
3.89
2.17
2.76
Token DI
3.81 + 0.26
5,56 + 0.34
5.83 = 0.13
3.93 + 0.03
Sentence ALDi*
3.35 # 0.09
3.89 + 0.25
5.16 = 0.13
4.15 # 0.1
Table C2: The (1) scores for the parallel MSA/DA corpora. TUN: Tunisian Arabic, MOR: Moroccan Arabic,
EGY: Egyptian Arabic, MGR: Maghrebi Arabic.
*; D' scores averaged across three fine-tuned models with different random seeds (30, 42, 50).
Segment
Sentence with English translation (ours)
ALDi (estimated)
Main speech
اايياا 2ي3 a fuetl التدلي ليين النيو التاعلنن صفاةة علي ف يقر الحاتي اب الو of علي واقهس تشروي الار الدول عني الأ تلامياتت من عملية اللاتت حلول
And we agreed the current circumstances endeavors all actors to bear their responsiblities by finding practical solutions and mechanisms to mitigate the impact the crisis on most affected countries.
Q&A - Gas
Jee والااا التنtيي الماالو إن الأمر عنا في تحفت ūn بعن تحمد إحنا الوول شنر eo be Bny فمر مs في خعاايي او حيين أككاه العاكف حنا يحس iA في العاتم م سكل تحديية الدلدد
0.41
spoke about this matter and that coordination and cooperation required between all countries of the world regarding this topic during my talk or speech at the Jeddah conference, specifically on the issue energy.
Q&A Q&A Human Rights
cnsiige go بلتا La
تمر التوو if فe pen لوو بيساا انتو ooe مونميين وححاا gعع بشسرر tim) سiin منر بيسووو بنا بنصتر عناز ملادية جييي و مسة as y wلde ببساوو أحدو عني موتيي منل إiدا يبااتبي يكم تمر تقممه وي ططي shاu الإتسايية و يخمي ولتاار
0.75
And are not interested in it because ask about it.. It is very important that you know this.. We are interested in it because we respect our people, and we love them, and these are not just words, we respect our people just as you respect your people.. and therefore we are not interested hecause you ask about No.. This is our moral, historical and humaniitaiia responsibility toward our people. This is one point.
Table D3: Three sentences of different estimated ALDi scores sampled from three segments of El-Sisi's speech the 22nd of July 2022 shown in Figure 4g.


Reviews:
[{"review_id": "dkEOEuq2Vw", "reviewer": "Reviewer_sVaX", "summary": "", "questions": "- In Table 1, how do you distinguish between the level of dialectalness and the level of formality? The distinction between the examples for the medium and high level of dialectalness is that the later is more informal (not necessarily more dialectal). I think higher level of dialectalness should be more related to how much of the given input is dialectal versus standard.", "limitations": "", "rating": 3, "confidence": 5, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the Reviewer sVaX for raising the question about the relationship between the “formality” and the “level of dialectness” concepts. We have thoroughly thought about this relationship while working on the paper, and dedicated a section to discussing why they are different, and why the \"level of dialectness\" is more suitable for Arabic. We decided later to remove it from our submission to avoid confusion to the reader. However, given that you have raised this very valid point, we will add it back. Please find a summary of our discussion of \"dialectness\" vs \"formality\" for Arabic below, which we will add to the camera-ready version of the paper.\n\nWhile we think the two concepts can be related, we will explain why we think the “level of dialectness” concept that we are introducing differs from “formality”, and is actually more suitable for Arabic:\n\n* During the literature review, we found that “formality” is a concept that has been widely studied, yet it does not generally have an agreed upon definition. Formality entails different stylistic dimensions such as serious-trivial, polite-casual and depends on the degree of shared knowledge [1]. (Pavlick et al, 2016) noticed that the annotator agreement scores are high for sentences at the extreme ends (i.e.: unambiguously formal or informal), but the agreement scores are lower otherwise [1].\n\n* In our paper, we define the “level of dialectness” as the degree by which a sentence diverges from the Standard Language. The “level of dialectness” is therefore topic-independent, and less ambiguous than “formality”.\n\n* To illustrate that the two concepts are different, and as demonstrated in Figure 4, we show that the level of dialectness of the political speeches of three Arab presidents is variable. Political speeches are typically considered to be formal, however, it is clear that the presidents might still use dialectal sentences in their speeches especially if they are spontaneously speaking, as in the speech visualized in Subfigure 4e. If you consider the following three sentences from this presidential press conference, which is a formal speech, you can see that the sentences have different levels of dialectness. However, if you define “formality” as a distinction between serious-trivial or polite-casual, then one can argue that all three sentences are highly serious and polite, which renders them all as being formal.\n\n\n| Predicted ALDi score  [0, 1]    | Sentence | Segment in Speech |\n| :---:        |    ----:   |          :---: |\n| 0 | واتفقنا على أن الوضع الحالي يفرض على كافة الفاعلين الدوليين التحلي بالمسؤولية لإيجاد حلول وآليات عملية تخفف من تداعيات الأزمة على الدول الأكثر تضررًا. | Main speech| \n| 0.41 | وأنا عايز أقول إن إحنا تحدث، يعني أنا تحدثت في هذا الأمر إن المطلوب التنسيق والتعاون بين كل دول العالم في ما يخص هذا الملف أثناء حديثي أو خطابي في مؤتمر جدة عن موضوع الطاقة تحديدًا. | Q&A - Gas | \n| 0.75 | وإحنا مش مهتمين بيه عشان أنتوا بتسألوا عنه.. مهم قوي إن إنتوا تعرفوا كدا.. إحنا مهتمين بيه عشان إحنا بنحترم شعوبنا، وبنحبها، ومش كلام، إحنا بنحترم شعوبنا زي ما إنتوا ما بتحترموا شعوبكم.. وبالتالي إحنا مش مهتمين عشان أنتوا بتسألونا عليه.. لأ.. ده مسؤوليتنا الأخلاقية والتاريخية والإنسانية تجاه شعوبنا. دي نقطة. | Q&A - Human Rights |\n\n* To the best of our knowledge, neither the \"formality\" nor the \"level of dialectness\" of **Arabic** text were studied before. Therefore, \nintroducing the AOC-ALDi corpus that is derived from the AOC corpus would be a great addition to the Dialectal Arabic resources, and would hopefully motivate more research in this direction. \n\n* To conclude, we agree that there is a relation between formality and the level of dialectness, since Arabic speakers tend to use MSA in formal situations, and their regional dialects in informal ones. However, an Arabic speaker can still use MSA and speak informally, or use their dialect and speak formally. Therefore, we believe that they are two different concepts that are not interchangeable.\n* Finally, we thank you again for raising this important point, and we will add this discussion (with its corresponding references) to the camera-ready version of our paper.\n\n**Ans.**: For the question about the examples in Table 1, we believe that they are consistent with our definition of “level of dialectness” as a measure of the divergence from the Standard language. For the “Medium” level of dialectness example, the verb “بسطنا” has an MSA etymology [2], [3], which is generally used to mean expand something, yet can also mean “cheering someone”. For the “High” level of dialectness examples, the verbs “شهيصنا” and “نغنجنا” do not have an MSA root, which makes them more divergent from MSA, and thus are higher in dialectness. \n\n\n***\n\n[1] Ellie Pavlick and Joel Tetreault. 2016. An Empirical Analysis of Formality in Online Communication. Transactions of the Association for Computational Linguistics, 4:61–74. (https://aclanthology.org/Q16-1005/)\n\n\n[2] https://en.wiktionary.org/wiki/%D8%A8%D8%B3%D8%B7\n\n\n[3] ​​https://www.almaany.com/ar/dict/ar-ar/%D8%A8%D8%B3%D8%B7/?c=%D8%A7%D9%84%D9%85%D8%B9%D8%AC%D9%85%20%D8%A7%D9%84%D9%88%D8%B3%D9%8A%D8%B7", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The studied problem (code switching in Arabic between the standard form and the dialectal variants) is an interesting one for Arabic language", "reasons_to_reject": "The proposed dataset only presents a marginal contribution over existing Dialectal Arabic resources and it is not clear that the task itself (i.e. level of dialectalness) is well defined since it conflates formality with dialectalness.", "questions_to_authors": "- In Table 1, how do you distinguish between the level of dialectalness and the level of formality? The distinction between the examples for the medium and high level of dialectalness is that the later is more informal (not necessarily more dialectal). I think higher level of dialectalness should be more related to how much of the given input is dialectal versus standard.", "paper_topic_and_main_contributions": "The paper tackles the problem of Dialect Identification in Arabic by annotating an existing corpus to add \"level of dialectalness\" labels to it. The existing dataset comprised comments crawled from several Arabic websites and annotated each comment as either being Dialectal or Standard Arabic along with the dialect (e.e. EGY, LEV, other) it belongs to. The authors fine-tune a BERT-based model in the dataset and compare the performance to different baselines (e.g., ones that measure the number of tokens present in or absent from dictionaries) and find that the fine-tuned model outperforms the baselines. While the paper tackles an important problem for Arabic, the proposed dataset only presents a marginal contribution over existing Dialectal Arabic resources.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "kC8YbyWmmj", "reviewer": "Reviewer_RifN", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We really thank Reviewer RifN for their interest in the paper and their positive feedback.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1- The authors introduce the concept of Arabic Level of Dialectness (ALDi), a continuous linguistic measure to capture the nuanced spectrum of dialect usage, shifting the focus from binary dialect identification to a continuous spectrum of dialectness.\n\n2-They establish the AOC-ALDi dataset, offering a robust foundation for evaluating this concept. The paper highlights that ALDi offers a more nuanced perspective compared to binary dialect identification systems. \n\n3- The paper's exploration of ALDi can inspire researchers to apply similar approaches to other languages.", "reasons_to_reject": "The paper could provide more details on how the proposed approach can be replicated, potentially offering guidelines for researchers interested in applying the ALDi framework to other languages", "questions_to_authors": "", "paper_topic_and_main_contributions": "1- The paper proposes a new approach to address the linguistic variations found in transcribed speech and user-generated text in Arabic, where a mixture of Modern Standard Arabic (MSA) and Dialectal Arabic (DA) is often present. While previous research has focused on binary Dialect Identification (DI) at the sentence or token level, the authors argue that this approach overlooks the nuanced spectrum of dialect usage that Arabic speakers perceive.\n\n2- To capture this nuanced variation, the paper introduces the concept of the Arabic Level of Dialectness (ALDi), which quantifies the degree of dialect usage in a sentence. They create a dataset called AOC-ALDi, derived from the Arabic Online Commentary (AOC) dataset, where each sentence is manually labeled with its corresponding ALDi level.\n \n3- The paper confirms that ALDi provides a more detailed and nuanced perspective compared to traditional DI systems, which categorize text as either MSA or DA. The authors present case studies to showcase how ALDi can reveal stylistic choices made by Arabic speakers in different contexts, highlighting its potential for sociolinguistic analyses.\n", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "foegGOvfES", "reviewer": "Reviewer_t5z9", "summary": "", "questions": "1. in Table 5 - ALDi scores are of three annotators? it's not clear from the caption.\n\n2. Was every sentence labelled thrice?\n\n3. Is the ALDi predicted label always in the set of {0, 1/3, 2/3, 1} or is a value between 0 and 1? \nAnd when you evaluate, do you compute the RMSE against the average or against the closest value of the multiple annotators?\n\n4. Figure A1 is not showing the sentences (broken image links appear)\n\n5. Have you considered using morphological analysis and disambiguation tools from Camel Lab... they exist for MSA and EGY... in principle you can use the ration of OOV against different analyzers.\n\n6. I assume you do not use MADAR in your annotations because of Baseline #2 which uses this data. perhaps clarify?\n\n7. Camel Tools dialect ID produces a vector of scores for MSA and dialects... why not compare against them? ", "limitations": "", "rating": 4, "confidence": 5, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are grateful to Reviewer t5z9 for thoroughly checking our paper and for their valuable questions and suggestions. Kindly find our responses below:\n\n* **Q1**: Yes, each sentence has three annotations. We will update the caption of Table 5 to make it clear.\n\n* **Q2**: Yes, every sentence was labelled thrice.\n\n* **Q3**: The fine-tuned model generates a continuous ALDi score in the range [0, 1]. For the evaluation, the RMSE is computed between the averaged human-assigned scores and the predicted value. We will make sure this is clearly mentioned in the captions of tables of figures (e.g.: Table 5, Figure 3) to avoid any confusion.\n\n* **Q4**: This screenshot is for the interface used by (Zaidan and Callison-Burch, 2011) [1], so we are afraid the images are not recoverable.\n\n* **Q5**: Thanks for the suggestion. While it would be one more good baseline to test, we think that the ability of this baseline to generalize to other lower-resourced dialects of Arabic would require building morphological analyzers for these dialects.\n\n* **Q6**: As mentioned in lines 185-196, we analyzed the “level of dialectness” labels that are already part of the Arabic Online Commentary (AOC) dataset which precedes the MADAR corpus. We will be sure that this is mentioned clearly in the final version of the paper.\n\n\n* Regarding the suggestion to **use the vector of scores from a Dialect Identification (DI) model** (+**Q7**), we tested the idea using the following equation **Estimated ALDi score(text) = 1 - MSAConfidenceScore(text)**, where MSAConfidenceScore is the confidence score that the Sentence DI assigns to the MSA label. The RMSE values shown in the table below show that using the confidence scores can slightly improve the baseline (RMSE 0.46 vs 0.49), however, it is still significantly worse than our proposed model (0.19). \n\n| Model | RMSE(Control Sentences) (N=2,127) | RMSE(Comments) (N=10,644) | RMSE(All) (N=12,771) |\n| ---: | :---: | :---: | :---: |\n| (old baseline) Sentence DI | 0.27 | 0.53 | 0.49 |\n| (suggested baseline) Sentence DI + Confidence scores | 0.27 | 0.48 | 0.46 |\n| (our proposed method) Sentence ALDi | 0.07 | 0.2 | 0.19 |\n\n  * Our intuition is that mapping the confidence scores that a Sentence DI model assigns to specific dialects vs. MSA is inadequate for estimating the ALDi score of a sentence. These confidence scores reflect the ability of the DI model to determine the specific dialect of a sentence. A model can be really confident about the dialect of the sentence, even if it only shows a single cue of dialectness that is quite distinctive for one dialect. However, a single cue of dialectness would not generally make the whole sentence highly dialectal (i.e.: the ALDi score should not be high).\nAgain, we thank you for the suggestion, and we will update the baseline in the paper to the one you suggested along with this discussion.\n\n\n\n\n\n\n***\n[1] Omar F. Zaidan and Chris Callison-Burch. 2011. The Arabic Online Commentary Dataset: an Annotated Dataset of Informal Arabic with High Dialectal Content. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 37–41, Portland, Oregon, USA. Association for Computational Linguistics. (https://aclanthology.org/P11-2007/)", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* The work presents a novel contribution to the field of Dialect Identification in general and Arabic specifically.\n* The effort is extensive\n* The paper is well written\n* The data is public", "reasons_to_reject": "* The agreement among annotators is only moderate.\n* One of the baselines used (#2, using Salameh/CamelTools) only uses the binary reading of the output; but the tool actually produces a vector of scores... why not compare against them?  Try the interface here: https://adida.abudhabi.nyu.edu/ to see the subscores.", "questions_to_authors": "1. in Table 5 - ALDi scores are of three annotators? it's not clear from the caption.\n\n2. Was every sentence labelled thrice?\n\n3. Is the ALDi predicted label always in the set of {0, 1/3, 2/3, 1} or is a value between 0 and 1? \nAnd when you evaluate, do you compute the RMSE against the average or against the closest value of the multiple annotators?\n\n4. Figure A1 is not showing the sentences (broken image links appear)\n\n5. Have you considered using morphological analysis and disambiguation tools from Camel Lab... they exist for MSA and EGY... in principle you can use the ration of OOV against different analyzers.\n\n6. I assume you do not use MADAR in your annotations because of Baseline #2 which uses this data. perhaps clarify?\n\n7. Camel Tools dialect ID produces a vector of scores for MSA and dialects... why not compare against them? ", "paper_topic_and_main_contributions": "The paper presents a new contribution to the area of Arabic Dialect Identification. Specifically the authors manually annotate a large corpus for a different degrees of dialectness at the sentence level and define what they name as the ALDi Score.   The intuition that dialectness in sentences can be represented on a continuum has been expressed in the past, but this work does a nice job of quantifying and studying and producing data sets to support this intuition.", "excitement": 4, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "N/A ", "Typos_Grammar_Style_and_Presentation_Improvements": "N/A"}, "extra_text": {}}]

Author rebuttal:

Meta-review: All reviewers agree that the paper contributes to an important research area, which is to add nuances of dialect usage by Arabic speakers in the existing work of Arabic Dialect Identification. While the idea of representing dialectness as a continuous variable isn't novel, the authors operationalize it and perform extensive annotations on the news articles and users' comments, where each sentence is annotated three times. Furthermore, one important highlight of the paper is to demonstrate the usefulness of the dataset for the sociolinguistic field, where the authors show that we can use models trained on their dataset to characterize the Arabic speaking stylistic choices (and even better than using vectors of scores from existing Dialect Identification models). During the post-rebuttal discussion, the authors also clarify the dimensional differences between dialectness and formality, and therefore this work contributes to the existing resources on formality and the binary representation of dialectness for Arabic dialect identification.
Decision: Accept-Main