Arabic Diacritics in the Wild: Exploiting Opportunities for Improved Diacritization (2024)

Abstract: The widespread absence of diacritical marks in Arabic text poses a significant challenge for Arabic natural language processing (NLP). This paper explores instances of naturally occurring diacritics, referred to as “diacritics in the wild,” to unveil patterns and latent information across six diverse genres: news articles, novels, children’s books, poetry, political documents, and ChatGPT outputs. We present a new annotated dataset that maps realworld partially diacritized words to their maximal full diacritization in context. Additionally, we propose extensions to the analyze-anddisambiguate approach in Arabic NLP to leverage these diacritics, resulting in notable improvements. Our contributions encompass a thorough analysis, valuable datasets, and an extended diacritization algorithm. We release our code and datasets as open source.

heading: 1 Introduction
text: Arabic orthography is infamous for its high degree of ambiguity due to its infrequently used optional diacritical marks. While other Semitic languages like Hebrew and Syriac use similar systems, Arabic has a richer inflectional space with case endings and other orthographic choices that make Arabic more complex. Interestingly, diacritical marks in Arabic are common in limited contexts where correct reading is a goal: holy texts, poetry and children’s books, as well as books for adult literacy and non-native learners. But in general reading contexts, for literate Arabic native speaker adults, diacritical marks are used frugally: ∼1-2% of words are partially diacritized (Habash, 2010). We refer to these as Diacritics in the Wild (WildDiacs).
In this paper, we follow in the footsteps of other researchers to investigate whether such precious occurrences can be exploited to help improve the quality of Arabic NLP tools (Diab et al., 2007; Habash et al., 2016; Bahar et al., 2023). We specifically focus on the Arabic diacritization task, which
is at once (a) a final target downstream application given the sparse and variable use of diacritics in Arabic; and (b) an enabling technology for other applications such as speech synthesis (Halabi, 2016).
While the percentage of WildDiacs is small, our guiding intuitions are that on large scales, these are objects worthy of study, and given the extra information provided in such contexts, we assume the writers who added them wanted to provide hints to support optimal reading, e.g., to avoid garden path sentences.
For a well-rounded exploration, we analyze and compare the diacritization patterns across six genres: news of multiple agencies, novels, children’s books, poetry, political/legal documents of the UN, and ChatGPT output (which sometimes introduces diacritics unprompted). Furthermore, we examine the diacritization patterns and choices in two commonly used datasets for evaluating Arabic diacritization: The Penn Arabic Treebank (Maamouri et al., 2004) and WikiNews (Darwish et al., 2017). We also develop a new annotated dataset that includes instances of partially diacritized words along with their full diacritization, which we define carefully, acknowledging different practices. And finally, we propose an extension to the analyzeand-disambiguate approach (Habash and Rambow, 2005; Pasha et al., 2014; Inoue et al., 2022) to improve the quality of its choices, and we evaluate on the data we annotated.
Our contributions are the following:
• We provide careful analysis and comparison of diacritization patterns in six genres of Arabic texts, shedding light on the needs and latent information in different Arabic genres. • We annotate a new dataset for studying maximal diacritization from partial diacritic signals (Wild2MaxDiacs), and we extend an existing dataset, WikiNews, to address unhandled phenomena (WikiNewsMaxDiacs).
14815
• We extend a hybrid (neuro-symbolic) algorithm for Arabic diacritization to make use of the existence of WildDiacs, and demonstrate improved performance.
Our code and datasets will be open-source and publicly available.1
The paper is organized as follows. Section 2 discusses Arabic diacritics. Section 3 reviews related work. Section 4 covers the datasets, including our annotated datasets, and WildDiacs usage statistics. Section 5 details the approach we build on and how we extend it. Section 6 presents the experimental setup, evaluation, and error analysis.

heading: 2 Arabic Diacritics
text: We present an overview of Arabic diacritics in terms of their form, function, and use.

heading: 2.1 Arabic Diacritic Forms
text: Arabic diacritics are zero-width characters that adorn Arabic letters to supplement Arabic’s Abjad orthography with additional phonological signals. There are many diacritical marks in the Arabic script: Unicode currently boasts 52 such symbols.2 However, the basic Tashkil (diacritization) set used in most Modern Standard Arabic (MSA) contexts includes nine symbols. See Figure 1 (a).3 In this paper, we focus on these MSA diacritics, which are all readily accessible by most Arabic keyboards.
Diacritic clusters can occur but are highly constrained. The Shadda (
∼) can combine with any one of the other diacritics, and none of them can combine with each other in MSA, except for the
1https://github.com/CAMeL-Lab/wild_diacritics 2https://unicode.org/charts/PDF/U0600.pdf 3Arabic transliteration is in the HSB scheme (Habash et al.,
2007): ’ Z, Ā @, Â @, ŵ ð, Ǎ @ , ŷ ø, A @, b H. , ℏ è, t H, θ H, j h. , H h, x p, d X, ð X, r P, z P, s , š , S , D , T  , Ď  , ς ¨, γ ̈ , f ¬, q , k ¼, l È, m Ð, n à, h ë, w ð, ý ø, y ø , ã , ũ , ı̃ , a , u , i , ∼ , . , á , Ä @.
Dagger Alif ( á), which can follow Fatha ( a). While in proper Arabic spelling the Shadda character should appear first in the string (and writing order) as it indicates doubling of previous consonant letter, it is important to note that a flipped order (e.g., Shadda after vowel) is impossible to detect visually. We find both orders in the wild. For example kat∼ab and kata∼b will always be rendered to appear as kat∼ab (I. J »).4

heading: 2.2 Arabic Diacritic Functions
text: Basic Phonological Mapping The diacritics primarily denote phonological information that supplements the Arabic Abjad orthography: vowel diacritics (Fatha, Damma, Kasra) indicate the presence of a short vowel; nunation ( áK ñ J K tanwiyn) diacritics indicate a short vowel followed by /n/; the gemination diacritic, Shadda, indicates doubling of the consonant letter it follows; the Sukun (silence) diacritic indicates that no vowel is present; and finally, the special elongation diacritic (aka Dagger Alif) indicates a long /ā/ vowel. See the red colored diacritics and their mapping to phonology in Figure 1 (b).
An important and a not so obvious detail about the use of diacritics is that even under maximal diacritization, some letters remain bare, i.e., with no diacritics, to indicate specific phonological information. Examples include (i) Elongation: the weak letters ( @ A, ð w, ø y) remain bare when used as helping letters to elongate short vowel diacritics (Fig. 1 (b)’s red letters); and (ii) Silence: letters that
4Most text rendering libraries that support Arabic implement the Arabic Mark Transient Reordering Algorithm (AMTRA) which normalizes the display of diacritic clusters (Pournader et al., 2021). Furthermore, many systems utilize Unicode Normalization Forms (UNFs) (Whistler, 2023b) which, among other things, order adjacent diacritics based on their combining class property (Whistler, 2023a). UNFs are important for string matching and lexicographic sorting despite diacritic order variability. These robustness-supporting mechanisms inadvertently allow inconsistent uses to coexist freely in the wild.
are phonologically elided or assimilated are marked by leaving them bare of diacritics (Fig. 1 (b)’s grey letters).5,6
Dimensions of Disambiguation We can categorize the functional use of diacritics based on the disambiguating information they provide to the reader, which includes lexical, morphosyntactic, and contextual phonological liaison (sandhi). For example, the word áÓ mn can be diacritized in different ways: áÓ min. ‘from’, á
Ó man. ‘who’, á Ó man∼a ‘he granted’, á Ó man∼ũ ‘a favor [in-
def. nom]’, among many others. All four instances show lexical diacritics. The last two words’ final diacritics indicate morphosyntactic features such as verb aspect-person-gender-number, and nominal case and state. In specific phonological contexts, some helping epenthetic vowels are introduced, and others may be elided. The typical epenthetic vowel is i, but there are other special cases: For example, min. ‘from’, has two additional forms: mina before words starting with the definite article (see Fig. 2), and mini in general epenthesis, e.g., é J K. @ á Ó mini Ab.nihi ‘from his son’.
5A very common example is the È l of the definite article È@ Al when followed by a Sun Letter – a coronal consonant with which the l assimilates, e.g., š and s in Fig. 1(b). A Shadda on the following letter indicates assimilated gemination.
6The Alif ( @ A) letter is used in word-initial positions as a vowel carrier (i.e., Alif Wasla, aka Hamzat Wasl). When the vowel is elided in context, the Alif is retained but kept bare, e.g., cases of grey @ A in Fig. 1(b). Quranic Arabic retains the Wasla diacritic @ Ä to mark this absence, but MSA does not use
it any more, which unavoidably leads to a minor ambiguity between silence and elongation. We internally model the Alif Wasla as it interacts with other diacritization decisions, but convert it to a bare Alif in the output. See Appendices E and F.

heading: 2.3 Arabic Diacritics in the Wild
text: The two main challenges concerning Arabic diacritics in NLP are incompleteness and inconsistency.
The Challenge of Incompleteness Arabic diacritics are quite often omitted, with around 1-2% of words in news text having at least one diacritic. Arabic’s rich templatic morphology, common default syntactic orders, and contextual semantic resolutions explain why educated Arabic readers can read Arabic with such a signal deficit. Not all texts and genres are equally devoid of diacritics. For Quranic Arabic, and to a lesser extent poetry, diacritics are almost always included to avoid any misreading of the texts. Similarly for children’s educational materials, diacritics are included to assist in learning. In this paper we study a number of genres to help understand how our approach will function under different conditions.
We define the terms undiacritized and dediacritized to refer to words with no diacritics, or stripped of all diacritics, respectively. We will use the term fully diacritized to refer to a number of standards that have been used to evaluate automatic diacritization processes (Fig. 2 ATB and WikiNews). We reserve the term maximally diacritized to the version we target in this paper as a higher standard of completeness (Appendix D). Partially diacritized refers to a state of in-betweenness where some diacritics are provided. WildDiacs are typically partial diacritizations.
The Challenge of Inconsistency We note two types of inconsistency in diacritic use. First, there are a number of acceptable variations that reflect different styles. These appear in the wild, as well as in commonly used evaluation datasets. Examples include (i) foreign names whose diacrtization
reflect local pronunciation such as AJ K A¢ QK. ‘Britain’ as b.riTaAniyaA or biriTaAniyaA; (ii) the inclusion of the Sukun (silence diacritic) at the end of utterance-final words; and (iii) the position of the Tanwiyn Fath before or after a word-final silent Alif or Alif-Maqsura such A K. A J» kitaAbãA or AK. A J» kitaAbAã ‘a book’. The second type are simply errors. Examples include (i) the Shadda appearing after the vowel diacritic, e.g., kata∼b instead of the correct kat∼ab ‘he dictated’; (ii) multiple incompatible diacritics on the same letter, e.g., ktAbuu instead of ktAbũ; (iii) diacritics in impossible positions such as word initial ı̃ktAb; and (iv) the correct diacritic is on the incorrect letter, e.g., ktiAb for kitAb.
To catch some of these inconsistencies, we developed a well-formedness check script and used it as part of this paper. Details are in Appendix B.
Despite their imperfections, WildDiacs are useful human annotations that not only aid other human readers, but can be exploited automatically.

heading: 3 Related Work

heading: 3.1 Roles of Diacritics in Arabic NLP
text: Diacritics play a significant role in a wide variety of NLP tasks, such as text-to-speech synthesis (Ungurean et al., 2008; Halabi, 2016), automatic speech recognition (Aldarmaki and Ghannam, 2023), machine translation (Diab et al., 2007; Alqahtani et al., 2016; Fadel et al., 2019b; Thompson and Alshehri, 2022), morphological annotation (Habash et al., 2016), homograph disambiguation (Alqahtani et al., 2019), linking lemmas across different lexical resources (Jarrar et al., 2018), language proficiency assessment (Hamed and Zesch, 2018), and improving text readability (Esmail et al., 2022; ElNokrashy and AlKhamissi, 2024).
Several attempts have explored the impact of varying degrees of diacritization in downstream tasks. Diab et al. (2007) and their follow-up work (Alqahtani et al., 2016) investigate the impact of various degrees of diacritization to identify the optimal amount of diacritics for better machine translation performance. Habash et al. (2016) observe a positive correlation between the degree of diacritization in the input text and the performance in the morphological annotation task.
Using naturally occurring diacritics in the input text is shown to be effective in the diacritization task itself. AlKhamissi et al. (2020) propose a model that accepts partially diacritized text during
decoding, demonstrating improved diacritization performance. With a similar motivation, Bahar et al. (2023) introduce a bi-source model that takes both characters and optional diacritics available in the input sequence. Our work differs from theirs in that no training is required to leverage the presence of diacritics in the input text, making our approach computationally efficient.

heading: 3.2 Datasets for Diacritization
text: Numerous datasets have been developed for diacritization in different variants of Arabic, such as MSA (Maamouri et al., 2004; Darwish et al., 2017), classical Arabic (Zerrouki and Balla, 2017; Yousef et al., 2019; Fadel et al., 2019a), and dialectal Arabic (Jarrar et al., 2016; Abdelali et al., 2019; ElHaj, 2020; Alabbasi et al., 2022). The source of datasets varies, including news, e.g., the Penn Arabic Treebank (ATB) (Maamouri et al., 2004) and the WikiNews dataset (Darwish et al., 2017), classical books (Tashkeela; Zerrouki and Balla, 2017), and poetry (APCD; Yousef et al., 2019). Among these resources, ATB and WikiNews are widely used as the standard benchmark datasets for diacritization in MSA.
Annotation conventions vary across datasets due to the lack of consensus in definitions, and may even be inconsistent within a dataset (Darwish et al., 2017). In this work, we thoroughly analyze and compare diacritization patterns in widely used datasets, highlighting the need for an evaluation set based on maximal diacritization—a refined definition of diacritization. We also provide a new annotated dataset based on this paradigm comprising 6,000 words across six different genres.

heading: 3.3 Automatic Diacritization
text: Approaches to automatic diacritization vary in task formulation, architecture choice, and the use of external resources. One line of work formulates diacritization as a single isolated task, e.g., a characterlevel sequence labeling problem (Zitouni et al., 2006, among others) and a sequence-to-sequence problem (Mubarak et al., 2019). Early efforts employ traditional machine learning models such as the maximum entropy classifier (Zitouni et al., 2006) and SVMs (Darwish et al., 2017). Recently, neural models have shown significant progress, such as LSTMs and their extensions (Abandah et al., 2015; Belinkov and Glass, 2015; Fadel et al., 2019b; Madhfar and Qamar, 2021; Darwish et al., 2021; Elmallah et al., 2024) and Transformer-based
models (Mubarak et al., 2019; Qin et al., 2021; Elmadany et al., 2023).
Another line of work addresses diacritization within a multi-task setup, jointly modeling diacritization with relevant tasks such as POS tagging (Habash and Rambow, 2005; Alqahtani et al., 2020, among others) and machine translation (Thompson and Alshehri, 2022). A large body of this kind has demonstrated the value of using external resources such as a morphological analyzer. They adopt an analyze-and-disambiguate strategy, where they generate possible analyses for each word with a morphological analyzer (analyze), then rank the analyses based on separately trained feature classifiers (disambiguate). This approach has been extensively explored using various architectures, including SVMs (Habash and Rambow, 2005, 2007; Roth et al., 2008; Pasha et al., 2014; Shahrour et al., 2015), LSTMs (Zalmout and Habash, 2017, 2019, 2020), and Transformer-based models (Inoue et al., 2022; Obeid et al., 2022). In this work, we present an extension to this approach where we utilize the presence of naturally occurring diacritics in the input text to improve the re-ranking of the analyses without additional training.

heading: 4 Diacritics: Data and Statistics

heading: 4.1 Datasets
text: We use eight pre-existing datasets (two fully diacritized and six partially diacritized), and we introduce two new maximally diacritized datasets that we make publicly available.
Fully Diacritized Datasets We include the WikiNews (Darwish et al., 2017) and ATB (Maamouri et al., 2004) datasets, which have been used extensively in past recording on Arabic diacritization (Darwish et al., 2017; Pasha et al., 2014; Mubarak et al., 2019). As mentioned in Sec. 2.3 and Fig. 2, WikiNews’ annotation guideline lead to a fuller diacritization than those of ATB’s essential guidelines. In Table 1, we use all the text of WikiNews, and the training data portion of ATB Parts 1-3 (Diab et al., 2010).
Partially Diacritized Datasets To study the variation in WildDiacs patterns in different genres, we targeted six datasets covering News, Poetry, Novels, Children’s books, UN, and ChatGPT. The Children and Novels set are from the Hindawi website7 (Al Khalil et al., 2018). The Poetry samples
7https://www.hindawi.org/
are from the Arabic Poem Comprehensive Dataset (APCD) (Yousef et al., 2019). The UN data is from the Arabic portion of the UN Corpus (Ziemski et al., 2016). The News data is from Arabic Gigaword (Parker et al., 2011), specifically from Asharq Alawsat (aaw) and AlHayat (hyt). We intentionally did not use the sources used in building the ATB corpus (afp, umh, nhr, asb) since ATB was used to train the baseline model we used in our experiments, and we did not chose Ahram (ahr) or Xinhua (xin) because of their very low percentage of WildDiacs. Finally, for ChatGPT, we use the dataset released by Alhafni et al. (2023) as part of their work on grammatical error correction, where they prompted ChatGPT-3.5 to correct undiacritized raw Arabic text and ChatGPT interestingly added seemingly random partial diacritizations. We randomly sampled 50,000 lines from Poetry, UN, and News due to their large size, while we used the whole datasets for the rest of the genres.8
Maximally Diacritized Datasets We introduce two new datasets. The first is the Multi-genre Wild Diacritization to Maximal Diacritization (Wild2MaxDiacs) dataset. Wild2MaxDiacs is composed of randomly chosen 6,000 words with wild partial diacritization, 1,000 from each of the six genres discussed above. For each selected word, given its full context, we provide a maximally diacritized version of it. The annotations were carried out by two Arabic native speakers. The annotators closely followed our definition of in-context full diacritization well-formedness. The annotations were passed through our well-formedness checks as an extra pass of validation and the words that failed our check were corrected. Wild2MaxDiacs is split into equal 3,000-word development (Dev) and Test sets.
The second dataset is WikiNewsMaxDiacs, a new version of WikiNews that we manually extended to our maximal diacritization, affecting 3.5% of its tokens. The main extensions are adding contextual diacritics and Dagger Alifs. We also, for the first time to our knowledge, annotated the dataset with valid alternative diacritizations (2.1% of tokens), such in the case of foreign names discussed above.

heading: 4.2 Statistics
text: Table 1 presents different diacritic usage statistics.
8All texts are processed with CAMeL Tools’ simple word tokenizer (Obeid et al., 2020).
Diacritization Completeness Across Genres Table 1 (c-d) highlight the great variation in use of diacritics across different genres. The fully diacritized datasets have almost complete diacritization by design. The columns in Table 1 are ordered to reflect the pattern of completeness (from left to right), except that ChatGPT, being artificial, is separated at the end. As expected, the degree of completeness in naturally occurring partially diacritized datasets is highest for Children text, followed by Poetry, Novels, UN documents and finally News. ChatGPT has an interestingly high level of WildDiacs comparable to Novels. We also observe that the number of diacritics per diacritized word correlates highly with the percentage of words with diacritics and the percentage of fully diacritized words—Table 1 (d-f). The low number of fully diacritized words for ATB and lower than perfect for WikiNews reflect the systematically missing diacritics in their annotations (see Section 2.3).9
Diacritization Patterns Across Genres Table 1 (g-o) shows the distributions of the nine diacritics (in Unicode order) across the datasets relative to the total number diacritics. Table 1 (p) presents the correlation of the diacritic distributions of each genre against the WikiNews distribution (fullest in diacritization). Interestingly, we note
9The main issues in ATB diacritization are (i) no a diacritic before Alif when indicating long vowel /ā/ , (ii) no Sukun to mark the definite article with moon letters and some foreign name consonant clusters, and (iii) no contextual diacritics.
that the genres with high percentages of diacritics are highly correlated with full diacritization; but as percentages drop, the ratio of lower frequency diacritics are more common (and the correlation becomes negative). This makes sense as it suggests WildDiacs fill in the gaps for the low frequency events. This is most evident when comparing News to WikiNews (same genre but different degrees of completeness): Sukuns drop in News to 1/20 of their WikiNews distribution; and Tanwiyn Fath increases by over 70 times. We also note that the ChatGPT correlation pattern with WikiNews is oddly different from all naturally occurring partial diacritization datasets. Finally, we note that the use of Dagger Alif is almost extinct across MSA; and it seems to mostly be present in the ATB tokenization. This is the only case where ATB has more details than WikiNews. We include Dagger Alif in our Maximal Diacritization.
Wild Diacritization Failures Table 1 (q) reports that almost all of the diacritized words pass our well-formedness check, with ChatGPT performing the worst by far. Table 1 (r) shows that the AlifTanwiyn order is split between two schools: ãA in WikiNews, Children and Novels; and Aã in ATB, Poetry, UN and News. ChatGPT produces a mix of the two approaches. And finally, according to Table 1 (s) the Shadda-Vowel order is almost perfectly stable, except in News and ChatGPT, where there is a lot of noise.
Functional Use of Wild Diacritization Finally, we report on the distribution of functional uses of WildDiacs in Table 2. We annotated the 3,000 words in the Wild2MaxDiacs Dev set across all six genres for six categories: lexical, case/state, person/gender/number/mood/aspect (PGNMA), passive verb, context diacritics, and errors. Lexical and case/state are the highest in frequency overall, followed by PGNMA then voice. The most dominant genres per category are: lexical, PGNMA and context (Children), case/state (News, 94% Tanwiyn Fath), passive voice (UN), and Errors (ChatGPT). The diversity and patterns of functional use focus are consistent with our expectations, although not previously reported to our knowledge.

heading: 5 Exploiting Diacritics in the Wild
text: Our basic approach to exploiting WildDiacs is based on the intuition that they are simply hints that we need to take (Bahar et al., 2023). We build on a hybrid (neuro-symbolic) analyze-anddisambiguate solution for Arabic diacritization implemented in CAMeL Tools (Obeid et al., 2020). While this open-source toolkit reports competitive results (Obeid et al., 2020, 2022), it has a number of limitations connected to its dependence on the-less-than-ideal ATB diacritization (training and tools). We present the basic approach followed by our modifications and extensions.

heading: 5.1 The Analysis & Disambiguation Process
text: Figure 3 shows the basic CAMeL Tools disambiguation process (in black components) and our extensions and modifications of it (in red components). First, for each word in the input sentence, a list of analyses is generated by a symbolic morphological analyzer. These analyses are comprised of a number of morphological features (e.g., part-ofspeech, gender, case), a number of lexical features (e.g., lemma, diacritized orthography), and a number of pre-computed statistical features (e.g., log probability of lemma). In parallel, a neural morphological tagger predicts a set of morphological features (a subset of features provided in analysis) for each word in context.10 Finally, the analyses are ranked using the tagger’s predictions by sorting on the number of matching morphological features per analysis, then breaking analysis ties with the following analysis features in order: the joint POSlemma log probability, the lemma log probability, and finally sorting lexicographically on the analysis diacritization, to ensure a final stable ranking.

heading: 5.2 Modifications & Extensions
text: Re-ranking with WildDiacs We modified the basic ranker to use Levenshtein insertion, deletion,
10We report here on the Unfactored BERT Disambiguator model (Inoue et al., 2022).
and substitution edits between input word and analysis diacritization. Empirically, our best setup is sorted with substitutions+deletions, followed by morphological features, substitution alone, deletion alone, POS+lemma log probability, lemma log probability, insertions, then diacritization lexicographic order. This order guarantees ideal behavior when the input is fully diacritized, and is backward compatible with the original ranker when the input is not diacritized. See more details in Appendix C.
Morphological Analyzer Modifications Our approach depends heavily on the choices produced by the morphological analyzers. The baseline analyzer we use is based on SAMA/Calima-Star (Graff et al., 2009; Taji et al., 2018) which follow the ATB full diacritization standard. As such, we make a number of changes to the morphological analyzer’s algorithm and database to accommodate maximal diacritization (Footnote 9). Examples include (i) adding a missing Fatha before Alif for long vowel representation, (ii) fixing the position in A/ý wordfinal Tanwiyn Fath, (iii) adding missing Sukuns in word final and medial locations, (iv) enhancing the database’s Alif Wasla coverage to include unmodeled cases, e.g., the determiner È @ Äal, and (v)
adding string flags to mark cases with allomorphic variants, e.g., áÓ min is marked as min%n (Section 2.3). For more details, see Appendix E.
Contextual Post Edit Extensions After analysis re-ranking, the top analysis for each token
is selected, and an array of all top analyses are passed through a new component that handles contextual post edits. The sequence of regex-based edits perform inter-word changes that depend on surrounding contexts and allomorphic flags introduced in the morphological analyzer. For example, the edits will transform the sequence I. m Ì' @ m%Ñ ë hum%m Äal.Hub∼u ‘they are love’ to I. m Ì'@ Ñ ë humu Al.Hub∼u. For more details, see Appendix F.

heading: 6 Evaluation

heading: 6.1 Experimental Setup
text: We evaluate our enhanced model in various settings to assess the impact of different extensions. Regarding the ranking algorithm, we compare the CAMeL Tools baseline (CT) with our extended re-ranking approach (CT++). Additionally, we conduct an Oracle evaluation to determine the upper limits of both versions by selecting the analysis closest to the gold diacritization for each word. For context modeling, we examine three scenarios: no modeling (None), modeling as standalone utterances (Solo), and full context (Full). Regarding the morphological analyzer database, we compare the Current CAMeL Tools database with our extended version (Extended). We assess performance across the Dev and Test sets of Wild2MaxDiacs’s six genres, without using any Wild2MaxDiacs data for training. Additionally, we evaluate the full system on WikiNewsMaxDi-
acs without partial diacritization to demonstrate the added value of our extensions. Our metric is strict word diacritization matching accuracy.

heading: 6.2 Results and Discussion
text: Wild2MaxDiacs Results Table 3 shows a result breakdown on Wild2MaxDiacs dataset. The results are consistent across genres, and both Dev and Test. Under All Dev, the morphological database modifications improved accuracy by 21.1% over the baseline model, and the contextual post-editing improved accuracy further by 10.2%. With all modifications applied, we achieve a 39.7% improvement in total, and we achieve close to 95.1% accuracy relative to Oracle.
We analyzed all the errors in the Dev set (n=354). As shown in Table 4, 55% are due to missing analyses in the morphological analyzers, and 22% are due to failures in morphological tagging. The rest of the errors are either wrong WildDiacs input (11%) or gold reference error (11%). There were very few cases of multiple different valid diacritizations (1%). These patterns were comparable across genres.
WikiNewsMaxDiacs Results Our modified system increases the strict matching accuracy of word diacritization from dediacritized input on original WikiNews (Darwish et al., 2017) from 47.1% (CAMeL Tools baseline) to 84.5% (our best system). The corresponding results on our newly annotated multi-reference WikiNewsMaxDiacs improve from 47.5% (CAMeL Tools) to 89.7% (our best system). 64% of the increase in the best system is due to improved basic gold reference in
WikiNews, and the rest is from the additional references. The improvements over the baseline show the added value of our modified components, although still below reported state-of-the-art systems (Elmallah et al., 2024); however, a direct comparison is not possible due to differences in training datasets and target diacritization standards. The increase in accuracy in the WikiNewsMaxDiacs set suggests more work is needed in the space of proper Arabic diacritization evaluation.

heading: 7 Conclusions and Future Work
text: In this paper, we conducted a detailed analysis of WildDiacs patterns across six different genres of Arabic text. We developed Wild2MaxDiacs, a new annotated dataset designed for evaluating models to exploit WildDiacs for maximal diacritization. Additionally, we extended the WikiNews dataset to WikiNewsMaxDiacs, aligning it more closely with our definition of maximal diacritization. We also enhanced an open-source toolkit, CAMeL Tools, to effectively utilize partial diacritization, resulting in improved performance.
In the future, we plan to enhance the morphological analyzer’s coverage, expand our research to include less explored genres like Classical Arabic texts, and investigate other wild text signals such as Hamza usage and dialectal spelling variations. All of the modifications and extensions will be integrated in a future version of CAMeL Tools. We also plan to study the effect of our system on downstream applications such as text-to-speech (Halabi, 2016; Abdelali et al., 2022) and Arabic romanization (Eryani and Habash, 2021).
Limitations
• We acknowledge that our downstream application evaluation focused solely on automatic diacritization, an important task in Arabic NLP with potential benefits for other applications.
• We acknowledge that the observations are limited by the selection of genres and sample sizes we studied.
• We acknowledge that the definition of maximal diacritization, as with full diacritization, is an open question and there may be different views on what is essential or not that we did not include or included, respectively.
Ethics Statement
• We do not violate any preconditions on the preexisting resources we use to our knowledge.
• The texts we release are either already out of copyright, creative commons, or allowable within fair use laws (short snippets).
• All new manual annotations were done by the paper authors and were compensated fairly.
• Like all NLP models, there is a chance that mistakes could be made by the system leading to unintended consequences. However, in this particular setup and problem, we are not aware of any sources of systematic bias or harm.

heading: Acknowledgements
text: We thank the anonymous reviewers for their insightful comments, and Kareem Darwish and Hamdy Mubarak for helpful conversations.

heading: A Experimental Details
text: Computational Resources All the models were run on a MacBook Pro with a 2.3 GHz Quad-Core Intel Core i7 Processor and 32 GB RAM. All the experiments were done within ∼11 CPU hours.

heading: B Well-formedness Check
text: Scope The well-formedness checking rules presented below focus on the validity of use of sequences of diacritics and specific letters. They do not guarantee that a word is correct linguistically, only that it has plausible maximal diacritization,
e.g., the non-word A¿ A ¾ » kuk∼aAkãA would pass
but the real word AK. A J » kuta∼AbAã would fail.
Word Diacritization Patterns A word is formed of an optional starting pattern, required recurring middle letter pattern and an optional ending pattern.
Starting patterns are ordered combinations of:
1. Conjunction: ð wa or ¬ fa
2. Preposition: H. bi, ¼ ka or È li 3. Definite article: È @ Aal. with Moon letters, or
◦È @ Aal◦∼ with Sun letters (◦)
4. Alif Wasla ( @ A) followed by a contextually dependent short vowel (see Context below)
The recurring middle pattern, repeated one or more times, is either of the following:
1. Any letter (except @ A) followed by ( ? (( | | ) | ) | ) ∼?((a|i|u)|aá)|\ .) 2. A long vowel: @ aA, ð uw, or ø iy. Ending pattern is one of the following:
1. A valid Tanwiyn cluster: any letter (except @ A) followed by ?( | | ) ∼?(ã|ı̃|ũ). A bare
Alif follows ã for all letters except the following final sequences: @ Â, Z @ A’, è ℏ.
2. Waw-of-Plurality: @ð uwA or @ ð w.A
Double Consonants Two consecutive letters with a Sukun each or a letter with Sukun followed by a letter with a Shadda are not allowed, unless at the end of the context.
Exceptions There is a small number of allowable exceptions, such as the name ‘Amr’ ending with silent ð w: ð Q Ô « ςam.raw and ðQ Ô « ςam.rı̃w.
Context The rules apply per word, but have access to surrounding context words to validate contextual diacritics. Punctuation marks and sentence boundaries constitute context delimiters. We assume that any unhamzated Alif @ A at the beginning of an input word is an Alif Wasla. There are two context diacritization well-formedness rules: (i) All words must end with a vowel representation (short or long) if followed by a word starting with an Alif Wasla; and (ii) Context-initial Alif Wasla, e.g., after punctuation or at beginning of sentence, must be followed by a short vowel diacritic.

heading: C Ranking and Re-Ranking
text: Ranking in Analysis and Disambiguation The ranking process consists of sorting the analyses using a ranking tuple for each analysis as a sorting key. For an analysis a, the ranking tuple ra is defined as ra = (Ma, Pa, La,Wa) used to sort the analyses where Ma is the number of morphological features matching the tagger’s prediction, Pa is the joint part-of-speech and lemma log probability of the analysis, La is the lemma log probability of the analysis and Wa is diacritized orthography of the analysis. While sorting analyses, their respective ranking tuples are compared item-wise only comparing subsequent items when the previous items are equal. Wa is used to sort analyses lexicographically when all the prior items are equal to ensure a stable ranking.
Re-ranking Modifications We extend this tuple with four new features. We first compute the Levenshtein distance between the original input word and Wa, both normalized to a UNF (see Footnote 4) with consistent Tanwiyn Alif order. We count the number of insert operations (Ia), substitution operations (Sa) and deletion operations (Da). We then create a new ranking tuple r′a = (Sa + Da,Ma, Sa, Da, Pa, La, Ia,Wa) which we use to re-rank the analyses. This new ranking tuple has the following properties:
• When an input word contains any diacritic, the analyses that change these diacritics the least are ranked higher. • When an input word has no diacritics, ra and r′a produce very similar rankings. • When an input word is fully diacritized, any exact matching analysis will have the top rank. • Deletions are preferred over substitutions since some diacritics can be removed while substitutions change the word meaning.
D Comparison of Different Diacritization Schemes
E Morphological Analyzer Modifications
F Contextual Edits

heading: Future Work
text: In the future, we plan to enhance the morphological analyzer’s coverage, expand our research to include less explored genres like Classical Arabic texts, and investigate other wild text signals such as Hamza usage and dialectal spelling variations. All of the modifications and extensions will be integrated in a future version of CAMeL Tools. We also plan to study the effect of our system on downstream applications such as text-to-speech (Halabi, 2016; Abdelali et al. , 2022) and Arabic romanization (Eryani and Habash, 2021). Limitations
• We acknowledge that our downstream application evaluation focused solely on automatic diacritization, an important task in Arabic NLP with potential benefits for other applications. • We acknowledge that the observations are limited by the selection of genres and sample sizes we studied. • We acknowledge that the definition of maximal diacritization, as with full diacritization, is an open question and there may be different views on what is essential or not that we did not include or included, respectively