A Context-Contrastive Inference Approach To Partial Diacritization (2024)

Abstract: Diacritization plays a pivotal role in improving readability and disambiguating the meaning of Arabic texts. Efforts have so far focused on marking every eligible character (Full Diacritization). Comparatively overlooked, Partial Diacritzation (PD) is the selection of a subset of characters to be annotated to aid comprehension where needed. Research has indicated that excessive diacritic marks can hinder skilled readers—reducing reading speed and accuracy. We conduct a behavioral experiment and show that partially marked text is often easier to read than fully marked text, and sometimes easier than plain text. In this light, we introduce Context-Contrastive Partial Diacritization (CCPD)—a novel approach to PD which integrates seamlessly with existing Arabic diacritization systems. CCPD processes each word twice, once with context and once without, and diacritizes only the characters with disparities between the two inferences. Further, we introduce novel indicators for measuring partial diacritization quality to help establish this as a machine learning task. Lastly, we introduce TD2, a Transformer-variant of an established model which offers a markedly different performance profile on our proposed indicators compared to all other known systems.1

heading: 1 Introduction
text: The Arabic language is central to the linguistic landscape of over 422 million speakers. It plays a pivotal role in the religious life of over a billion Muslims (Mijlad and El Younoussi, 2022). As in other impure abjad writing systems, the Arabic script omits from writing some phonological features, like short vowels and consonant lengthening. This can affect reading efficiency and comprehension. Readers use context from neighbouring words, the domain topic, and experience with the
1 Demo: https://huggingface.co/spaces/bkhmsi/Par tial-Arabic-Diacritization
language structure to guess the correct pronunciation and disambiguate the meaning of the text.
The Arabic NLP community has noticeably focused on the task of Full Diacritization (FD)—the modeling of diacritic marks on every eligible character in a text (for example: Darwish et al., 2017; Mubarak et al., 2019; AlKhamissi et al., 2020). This is especially useful in domains where ambiguities are not allowed, or where deducing the correct forms might pose challenges for non-experts. Such domains may include religious texts, literary works like poetry, or educational material.
There are benefits for human readers, like facilitating learning. However, prior research suggests that extensive diacritization can inadvertently impede skilled reading by increasing the required processing time (Taha, 2016; Ibrahim, 2013; AbuLeil et al., 2014; Midhwah and Alhawary, 2020; Roman and Pavard, 1987; Hermena et al., 2015). Nevertheless, diacritics are important morphological markers, even when excessive, and may benefit automated systems in language modeling, machine translation (MT), part-of-speech tagging, morpho-
89
logical analysis, acoustic modeling for speech recognition, and text-to-speech synthesis. For an example in MT, see Fadel et al. (2019); Habash et al. (2016); Alqahtani et al. (2016); Diab et al. (2007).

heading: 1.1 Contributions
text: Context-Contrastive Partial Diacritization We propose a novel method for Partial Diacritization (PD) which seamlessly utilizes existing Arabic FD systems. We exploit a statistical property of Arabic words wherein readers can guess the correct reading of most unmarked words with minimal context. To select the letters to mark, CCPD sees each word twice: (1) within its sentence context, and (2) as an isolated input with no context. The two predictions are combined to retain only those diacritics which present comparatively new information that may aid reading comprehension (Section 4).
Human Evaluation We conduct a behavioral experiment to compare ease of reading for text with all, some, or no diacritics. Diacritics are selected via a neural model mask to simulate native partial marking. The results support prior work on the effect of different degrees of diacritcs on reading efficiency and comphrension, and is the motivation for this work (see Section 8).
Performance Indicators We introduce a set of automatic indicators to gauge the performance and usefulness of our method on several public mod-
els, in light of the scarcity of supervised labeled partially diacritized datasets (see Section 5).
Transformer D2 And last, we present the TD2 model, a Transformer variant of D2 (AlKhamissi et al., 2020), which shows markedly improved Partial DER performance at 5.5% compared to 11.2%, even while marking a larger percentage of text at 24.6% compared to 6.5% for D2 (see Section 6.1).

heading: 2 Motivation
text: Previous research underscores the substantial influence of diacritization on the reading process of Arabic text. Skilled readers exhibit a tendency to read highly diacritized text at a slower pace compared to undiacritized. The effect on reading speed and accuracy has been substantiated across numerous studies involving diverse age groups and linguistic backgrounds (Taha, 2016; Ibrahim, 2013; AbuLeil et al., 2014; Midhwah and Alhawary, 2020; Roman and Pavard, 1987; Hermena et al., 2015; Hallberg, 2022). In addition, studies monitoring eye movements during reading link the increased reading times for diacritized text to an increase in fixation frequency and duration—a key indicator of the word identification process (Roman and Pavard, 1987; Hermena et al., 2015). Partial Diacritization may thus lead to a better reading experience for a wide range of readers, including in cases with dyslexia or visual impairments.
سردی سرّدی سرّدُیسردُی سردَُی س ِرّدُی سُ ِرّدَُی
No Diactitization
Full Diactitization Partial Diactitization

heading: 3 Arabic Preliminaries
text: Diacritics in Modern Arabic Orthography The inventory of diacritics used in modern typeset Arabic comprises at least four functional groups. First are h. arakāt and sukūn, which indicate vowel phonemes or their absence. Then tanwı̄n and shaddah indicate case-inflection morphemes and consonant lengthening. There are other marks, e.g. for recitation features in the Qur’an. Only the marks in Table 2, part of modern Arabic orthography, are discussed here. H. arakāt are vowel diacritics that provide vowel information. Sukūn indicates the absence of a vowel, which indicates consonant clusters or diphtongs. Tanwı̄n diacritics indicate the phonemic pair of a vowel and the consonant /n/ at the end of words, serving case inflection purposes. These diacritics play essential roles in Arabic orthography and pronunciation (Hallberg, 2022).
Modes of Diacritization. Since diacritics are optional and their usage can vary widely—practical patterns have emerged. Hallberg (2022) identifies seven modes of diacritization. These modes can be ordered based on the quantity of diacritics used, ranging from no diacritization to complete diacritization. Deeper levels, like complete diacritization, are less frequently used than shallower levels (no diacritization, and so on). Examples in Figure 2.

heading: 4 Methods

heading: 4.1 Full Diacritization Models
text: We utilize deep neural sequence models for character diacritization. We test the pretrained models D2, Shakkelha and Shakkala (AlKhamissi et al., 2020; Fadel et al., 2019; Barqawi, 2017). Further, we introduce the TD2 model: a Transformer variant of the LSTM-based D2. A description of TD2 and the D2 architecture can be found at Section 6.1.

heading: 4.2 What Context Do Models See?
text: Context extraction. Let ctxt(x;X) contextualize x from X. Then the word and letter context extractors are functions which use the surrounding words and letters around positions i and j.
Let the sub-sequence si,T contain all the words in a segment of length 2T s.t. si,T = {wt | t ∈ i± T} as used in training D2/TD2.
{ wi } T = ctxt(wi; {w . . . ∈ si,T }) (1)
{ ℓj ∈ wi } = ctxt(ℓj ; {ℓ . . . ∈ wi }) (2)
Words around position i
Letters in Word i
Thus {wi }T refers to a segment containing the word i, while {ℓj ∈ wi } refers to the intra-word context for letter j in word i.
Contextual Prediction sees the full segment.
fsent(i, j) = f ( {wi }T , {ℓj ∈ wi } ) (3)
Single Word Prediction sees only the current word2 at position i. Note the wi instead of {wi}T .
fword(i, j) = f ( wi , {ℓj ∈ wi } )
(4)
Thus the two modes are defined as follows: fword is single-word application, while fsent uses sentence context. Both use the same model f .
4.3 Context-Contrastive Partial Diacritization The function fword takes each word in isolation. If its prediction is correct, we assume that this reading is common and easy to guess by a human reader. By contrast, fsent utilizes word context within a sentence. If a word has multiple possible readings which the sentence disambiguates, we expect fsent with sentence context to out-perform fword.
Following from these premises, we propose the following method to combine both sources of information to diacritize text partially, with justification.
CCPD: Algorithm Using fsent and fword predictions for a word and letter (wi, ℓj), the system assigns or omits a diacritic according to CCPD(i, j):
CCPD(i, j) = { ysent(i, j) ← mark(i, j), ∅ ← otherwise
(5)
2 Sentences are split on spaces. Words are not tokenized.
In hard or disagreement mode, the letters where inferences with and without context agree are left bare (regardless of the correctness of the prediction; unknown during inference). Otherwise, the contextual prediction ysent is returned.
mark hard
(i, j) := ysenti,j ̸= ywordi,j (6)
In soft or confidence mode, only the letters where the logit ỹsent exceeds the logit ỹword by a margin θ receive a diacritical mark.
mark soft
(i, j) := ỹsenti,j > θ + ỹ word i,j (7)

heading: 5 Performance Indicators
text: To measure performance on the partial diacritization task, we propose four indicators as approximate gauges for model alignment with human expectation. These indicators aim to address the challenge of the limited availability of high-quality partially diacritized test datasets, providing valuable insights into the model’s capabilities.
Indicators may use the marked (DS) and unmarked (DU) subsets of the corpus (D).
DS = { i, j ∈ D | mark(i, j) } selected DU = { i, j ∈ D | ¬mark(i, j) } unmarked (8)

heading: 5.1 SR: Selection Rate
text: SR (D) = |DS||D| (9)
SR is the proportion of characters assigned a diacritic by Function (5). Literature shows partial diacritization hovers around 1.2%–9.5% coverage in some professionally published books (Hallberg, 2022), while other research suggests that deliberate partial diacritization by native speakers results in a rate around 19%–26% (Esmail et al., 2022).

heading: 5.2 Scoped Diacritic Error Rates DER(f, D)
text: DER( f , D ) = 1 | D |
∑
i,j∈D
1( f ̸= gt ) i,j
(10)
DER in traditional literature is the ratio of diacritics erroneously predicted over all eligible letters in a corpus. Trivially, we parameterize DER by the system f and corpus (subset) D.
Predictor True labeler
(Subset of) letters Word i, char j

heading: 5.2.1 P-DER: Partial Diacritic Error Rate
text: PDER(D) = DER( fsent , DS ) (11) We calculate the DER on DS, which includes all characters in the corpus assigned diacritics by CCPD, following Function (5).
Contextual prediction Marked letters

heading: 5.2.2 B-DER: Basic Diacritic Error Rate
text: By the intuition in Section 4.3 regarding the noncontextual predictions of fword, we calculate the DER on the whole corpus D and use B-DER as a proxy for human error on plain unmarked text.
BDER (D) = DER( fword , D ) Non-Contextual prediction All letters (12)

heading: 5.3 RE-DER: Reader DER
text: We combine the Partial DER and Basic DER into one general measure of the total error in a partially diacritized text. The Basic DER indicator is used as a proxy which may be taken as an upper bound for mistakes made by an experienced reader. Formally, this indicator incorporates the error of the non-contextual model for the unmarked text, and of the contextual-model for the marked text (because the system explicitly annotates it). Thus RE-DER (Reader DER) is:
HEDER (D) = (SR) × DER(fsent,DS) + (1− SR)× DER(fword,DU)
(13)

heading: 5.4 SU: Signal Utilization
text: How well does CCPD utilize its marked subset to disambiguate the text? We gauge how informative the added diacritics are by measuring the change in DER within the marked subset of text. SUspans [−1, 1] where positive values denote improvement over no annotation. SU is defined as:
SU(D) = BDER − HEDER
SR (14)
E.g. an SU=50% shows that half of the produced annotations clarify ambiguities (lower RE-DER).

heading: 6 Experimental Modeling

heading: 6.1 TD2: Transformer D2
text: The TD2 model is a Transformer adaptation of D2 (AlKhamissi et al., 2020). It comprises two encoders of 2 layers each, for tokens and characters. Both models use the same architecture. The
feature and intermediate widths are (768, 2304). The token model is initialized from the pretrained weights of layers 1-2 of CAMeL-Lab/bert-basearabic-camelbert-mix-ner (Inoue et al., 2021) as provided by HuggingFace3. The character model is initialized from layers 3-4 of the same pretrained weights.

heading: 6.2 SP: Single-Pass Inference
text: For recurrent (non-Transformer) models like D2, we also test inference without segmentation and majority voting. Each sentence is passed in full the model, once. This can save inference time (by avoiding overlapping windows), but sacrifices some reliability (as the model had been trained on smaller window sizes).

heading: 6.3 Other Models
text: Shakkala (Barqawi, 2017) and Shakkelha (Fadel et al., 2019) are LSTM-based models that view sentences as a sequence of characters. In contrast, D2 and TD2 view sentences as word sequences.

heading: 7 Results

heading: 7.1 Full Diacritization Performance
text: Following prior work, we report the Diacritic Error Rate (DER) and Word Error Rate (WER) including
3 HuggingFace: Wolf et al. (2020)
case-endings which are located at the word’s end, usually determined by the word’s syntactic role. Predicting these diacritics is more challenging compared to core-word diacritics, which specify lexical properties and lie elsewhere within the word.

heading: 7.2 Partial Diacritization Performance
text: Intuition Overall it is desirable to observe: (1) a natural SR value close to native human annotation rates; (2) a low B-DER close to the SR, signifying a capable model which allows a clean partial annotation of only the hardest letters, (3) a low P-DER to signify high accuracy in the committed diacritics, and (4) a low RE-DER which gauges the overall expected reading experience given the partial annotation and the expected guessing error. Overall, this corresponds to a balance between a high SU, a natural SR, and a low P-DER, roughly ordered.
Model results Notice that the recurrent D2 is conservative in its selection with SR at 6.46% versus TD2’s 24.61%. TD2 under-performs in noncontextual mode with B-DER at 25% compared to 7.1% by D2. Counter-intuitively, this leads it to a higher SU at 91.5% as its contextual mode corrects 22.5 of every 25 errors its non-contextual mode commits. Both systems agree that much of the text (at least 75%) is reasonably guessable by the reader without annotation, by taking B-DER as an upper-bound
on human guess error. We may claim that between 75% and 93% of characters require no diacritization to disambiguate (the extremes of the two systems). Both D2 and TD2 correct most of their B-DER error via CCPD, implying that most letters which had errors in non-contextual mode were accurately selected by CCPD and correctly predicted by fsent, leading to an RE-DER < B-DER.
Soft marking SR can be tweaked in soft selection mode. Let’s analyze the model with SR=15.2% (Table 3) which provides a low P-DER at 5.1% (few diacritics added are erroneous) and a low RE-DER at 2.3% (around 4.8 out of 7.1% expected errors are corrected via the added diacritics). Notice the low SU (utilization) however, as only 31.6% of the annotated letters contribute to the drop in error. Assuming that the selected letters are the hardest to guess for a reader, this suggests that D2-SP models can disambiguate the text by 39.4% to 71.8% (relative improvement in RE-DER/B-DER) by marking only 5.9% to 21.8% of it. While RE-DER improves, P-DER and SU worsen, suggesting that increased annotation may give only marginal improvement in overall readability if the final model output is not perfectly clean.

heading: 8 Behavioral Experiment on Partial Diacritization
text: We start with the hypothesis that reading partially diacritized text may be easier than reading fully diacritized text, while providing some benefit in disambiguation or ease-of-reading.
To test this hypothesis, we conduct a behavioral experiment using machine-predicted partial diacritization masks. The diacritical marks themselves are the true labels. This is to ensure that the presented text is accurate, even if the model output is sub-optimal. The question is then: Does CCPD using fsent and fword select useful letters to be diacritized? See the results in Figure 4.

heading: 8.1 Behavioral Experiment Setup
text: Demographics We utilize the PsyToolkit online platform4 to conduct a behavioral experiment aimed at assessing the impact of different levels of diacritization (modes) on reading speed and accuracy. Data was gathered from a group of 15 participants, covering various age groups and native speakers of different Arabic dialects. The majority
4 PsyToolkit: Stoet (2010, 2017)
of participants (10) fell within the 25–40 age group; 2 were below 25, and 3 were above 40. All participants spoke at least 2 languages, while 6 spoke 3 or more. Eight participants spoke either Gulf (4) or Maghrebi (4) Arabic natively, and 7 spoke Egyptian (2), Levantine (2), or Sudanese (2). One did not report a native dialect. All participants reported being native speakers of Arabic.
Dataset To measure the impact of diacritics, we select 30 sentences from various domains in the WikiNews testset (Darwish et al., 2017). Each sentence is presented in three variants with Zero/Partial/Full diacritics. The Partial variant is produced using our CCPD algorithm and the D2-SP model in hard selection mode to mask out easy or guessable ground truth diacritics.
Data Splits The data is split into 3 buckets of 10 sentences each. We create 3 test sets such that the buckets are rotated and each appears exactly once in any mode (Zero/Partial/Full). For example, the first bucket would appear in the test sets A, B, C in its Full, Zero, and Partial variants. This is done to ensure that any participant sees a sentence exactly once, to avoid biasing their rating by repeated exposure. This ensures also that each sentence is seen equal times in each mode.
Timing and Scoring Participants are shown each sentence for a few seconds proportional to the word count of the sentence: 1/4 |words|. Then the participants are prompted to rate their comprehension by a score from 1 to 5, where 1 indicates difficulty and 5 indicates ease in understanding the sentence. The design intention is to measure the reading experience of a native speaker when scanning a nontechnical text in Modern Standard Arabic.

heading: 8.2 Findings
text: Figure 4 illustrates the per-mode average selfreported reading comprehension scores, normalized by the score of the no diacritization (Zero) mode. The results indicate that Full diacritization hinders reading accuracy when participants are provided with a limited amount of time to read the text, aligning with findings discussed in Section 2. In contrast, the Partial diacritization mode sometimes enhances reading comprehension performance compared to the Zero mode.
The data from this behavioral experiment will be available on the paper’s GituHub repository5. 5 GitHub: munael/arabic-partial-diacritization

heading: 9 Related Work
text: The literature has dealth with both types of diacritization: Full (FD) and Partial (PD). The majority has focused on FD. In PD, a distinct approach that seeks to augment reading comprehension by incorporating only the minimal requisite diacritics, we note a few works focus (Almanea, 2021; Mijlad and El Younoussi, 2022). Other works take a morphological analysis approach (Obeid et al., 2022, 2020; Alqahtani et al., 2016; Shahrour et al., 2015; Habash and Rambow, 2007).

heading: 9.1 Full Diacritization
text: Non-Neural Methods This class of methods combines linguistic rules with non-neural modelling techniques, such as Hidden Markov Models (HMMs) or Support Vector Machines (SVMs), which were widely employed over a decade ago. For example, Elshafei et al. (2006) utilize an HMM to predict diacritization based on bigram and trigram distributions. Bebah et al. (2014) extract morphological information and utilize HMM modeling for vowelization, considering word frequency distribution. (Shaalan et al., 2009) combine lexicon retrieval, bigram modeling, and SVM for POS tagging, addressing inflectional characteristics in Arabic text. Darwish et al. (2017) operate diacritization in two phases, inferring internal vowels using bigrams and handling case endings via an SVM ranking model and heuristics. Said et al. (2013) follow a sequence-based approach, involving autocorrection, tokenization, morphological feature extraction, HMM-based POS tagging, and statistical modeling for handling OOV terms. Zitouni and Sarikaya (2009) approaches the problem using maximum entropy models.
Neural Methods Recent literature has focused more on neural-based systems. Belinkov and Glass (2015) were the first to show that recurrent neural models are suitable candidates to learn the task entirely from data without resorting to manually engineered features such as morphological analyzers and POS taggers. AlKhamissi et al. (2020) used a hierarchical, BiLSTM-based model that operates on words and characters separately with a crosslevel attention connecting the two—enabling SOTA task performance and faster training and inference
compared to traditional models. Many prior works have used recurrent-based models for the FD task (Al-Thubaity et al., 2020; Darwish et al., 2020; Gheith Abandah, 2020; Fadel et al., 2019; Boudchiche et al., 2017; Moumen et al., 2018), others used the Transformer architecture (Mubarak et al., 2019), while others used ConvNet architectures (Alqahtani et al., 2019b).
Hybrid Methods Some works have combined neural and rule-based or other methods to improve Arabic diacritization (Alqahtani et al., 2020; Abbad and Xiong, 2020; Darwish et al., 2020; Alqudah et al., 2017; Hifny, 2018).

heading: 9.2 Partial Diacritization
text: Similar to this work, previous research has explored the selective diacritization of Arabic text. Alnefaie and Azmi (2017) harnessed the output of the MADAMIRA morphological analyzer (Pasha et al., 2014) and leveraged WordNet to generate word candidates for diacritics. This work focused on resolving word ambiguity through statistical and contextual similarity approaches to enhance diacritization effectiveness. Alqahtani et al. (2019a) focused on selective homograph disambiguation, proposing methods to automatically identify and mark a subset of words for diacritic restoration. Evaluation of various strategies for ambiguous word selection revealed promising results in downstream applications, such as neural machine translation, partof-speech tagging, and semantic textual similarity, demonstrating that partial diacritization effectively strikes a balance between homograph disambiguation and mitigating sparsity effects. Esmail et al. (2022) employ two neural networks to predict partial diacritics—one considering the entire sentence and the other considering the text read so far. Partial diacritization decisions are made based on disagreements between the two networks, favoring the prediction conditioned on the whole sentence.

heading: 10 Discussion
text: In this work, we have presented a novel approach for partial diacritization using context-contrastive prediction. The task is motivated by a large body of literature on the impact of diacritization coverage on the reading process of Arabic text, and the scarcity of research into systems to automate it.
Context-Contrastive Prediction By considering contextual information alongside non-contextual
predictions, our approach builds on the role of diacritics in text, particularly in the context of the Arabic script as used by humans where accuracy is not the only consideration. The result is a simple, efficient, and configurable method which can be integrated with any existing diacritization system. It opens doors to enhanced diacritization accuracy and selection. The indicators we introduce provide valuable signals for evaluating such advancements.
The Effect of Diacritics on Reading A significant motivation for our work rests upon prior research that into the substantial influence of diacritization on the reading process of Arabic text. Multiple studies have consistently shown that skilled readers tend to read extensively diacritized text at a slower pace than undiacritized text. This phenomenon, supported by research involving diverse age groups and linguistic backgrounds, has significant implications for reading accuracy and comprehension (Taha, 2016; Ibrahim, 2013; Abu-Leil et al., 2014; Midhwah and Alhawary, 2020; Roman and Pavard, 1987; Hermena et al., 2015; Hallberg, 2022). Other studies also suggest that diacritization not only affects reading speed but also directly impacts reading accuracy (Roman and Pavard, 1987; Hermena et al., 2015). In this work, we complement this large body of research by conducting a behavioral experiment that utilized our CCPD approach to partially diacritize news headlines from the WikiNews test set. Our results show that partially diacritized text is easier to read than fully diacritized text, and in some cases can lead to better understanding than no diacritization at all.
Implications An exciting potential for this work is optimizing the reading experience for different groups of readers with different needed levels of diacritization, including people with dyslexia and visual impairments. Since diacritization plays an important role in disambiguating the meaning of the text, it is crucial to intelligently select the ones that aid readability and comprehension of Arabic text without excessive marking; making it accessible and accommodating to a broader audience. This aligns with the broader goals of inclusivity and accessibility in NLP applications.
Future Directions Further research is needed to gather high-quality benchmark data for partial diacritization, to enable more traditional and direct performance metrics alongside the indicators we propose. One promising direction is evaluating the
method on other languages which utilize diacritical marks in their orthographies.

heading: 11 Conclusion
text: In conclusion, our CCPD approach for partial diacritization using context-contrastive prediction contributes to the field of diacritization in an area with far-reaching implications for enhancing the reading experience of Arabic text. Grounded in previous research on the influence of diacritization on reading and further supported by behavioral experiments conducted in this work—our approach paves the way for advancements that benefit readers of diverse backgrounds and abilities. Since our approach integrates seamlessly with existing Arabic diacritization systems, it can be used by models trained on domain-specific text as well. We also propose a battery of performance indicators to gauge the competence of partial diacritization systems using fully-diacritized test data to mitigate the lack of publicly available benchmarks. Finally, we introduce TD2—a Transformer adaptation of the D2 model which offers a different performance profile as shown by our proposed indicators. As Arabic NLP continues to evolve, our approach serves as a promising direction for enhancing diacritization and its impact on text accessibility and comprehension.
Limitations
The method presented in this work is tested on a single human language. While we believe it should be able to generalize to other languages that need similar diacritics restoration, its utility needs further observation. In particular, it may be the case that partial diacritization patterns in languages besides Arabic are different in such a way that the premise of this work no longer holds (wherein ease of guessing is regarded as a major factor in omitting a diacritical mark).
In addition, this method is deliberately kept simple, which builds on an over-simplified view of the human task. Readers guess diacritics when reading, but they may do so while incorporating context via some simple or fast mechanisms. Whether that is at a deep level equivalent to guessing the reading of a single word in isolation remains to be seen. Our method uses non-contextual application, fword, as a proxy for this process, which is likely relatively close in performance to the same mechanism in a human. Nevertheless, we make no claim that this
is indeed the natural mechanism, nor that the proxy exhibits identical performance distribution to the natural mechanism.

heading: A Experimenting with [Transformer] D2
text: A.1 Datasets In this work, we train on data from the Tashkeela (Fadel et al., 2019) and Ashaar (Alyafeai et al., 2023) corpora. We test on the Tashkeela testset. The Tashkeela corpus has been collected mostly from Islamic classical books (Zerrouki and Balla, 2017) and contains mostly classical Arabic sentences. Ashaar is a corpus of Arabic poetry verses covering poems from different eras. Table 4 shows the number of tokens for the Tashkeela and Ashaar6 datasets.
A.2 Majority Voting using a Sliding Window Following prior work that utilizes an overlapping context window approach with a voting mechanism to enhance diacritic prediction for individual characters (Mubarak et al., 2019), we segment each input sentence into multiple overlapping windows.
6 The reported numbers reflect dataset cleaning to keep only Arabic letters and diacritics.
We present each segment to the model separately. This approach has proven effective, as localized context often contains enough information for accurate predictions. We use similar segmentation parameters to AlKhamissi et al. (2020)
Training For the training and validation sets we use a window of 10 words and a stride of 2.
Inference The same character may appear in different contexts, and therefore potentially result in different diacritized forms. We implement a popularity voting mechanism to narrow down the prediction. When a tie arises, we randomly select one of the top options. Testing is done with a sliding window of 20 words at a stride of 2.
A.3 Datapath of D2, TD2 The token model’s output is averaged per word to result in exactly one feature vector (instead of one for each sub-word). This aggregated vector zw is concatenated with the character embedding of each character in the word and down-projected to the model feature size, to get character input xc,w. The character encoder transforms the inputs {xc,w | c ∈ chars(w)} into character feature vectors zc,w which are passed to the final classifier.
Training The full model is tuned using the AdamW optimizer (Loshchilov and Hutter, 2019) with 0.2 dropout and 5×10−4 LR, and follows a linear schedule of 500 warmup steps and 10, 000 total training steps. The best checkpoint at 1, 000 step intervals is picked by the combined Tashkeela and Ashaar dev sets.

heading: B Demo and Examples
text: We developed an online Demo on Huggingface7 which allows choice between Full Diacritization, and Partial Diacritization with Hard and Soft modes. See Figures 5a and 5b. The demo supports the D2/TD2 models (trained on Tashkeela and Tashkeela/Ashaar). Since they focus mostly on classical Arabic text, users are advised not to anticipate optimal performance when applying this
model to Modern Standard Arabic (MSA) or informal Arabic dialects.
B.1 More Partial Diacritization Examples The following examples are lines taken from a poem by Rami Mohamed (2016) called “We Will Stay Here”. We applied the models as indicated in the System column. The input text included no diacritics at all. See Table 5.
7 huggingface.co/spaces/bkhmsi/Partial-ArabicDiacritization

heading: Future Work
text: Future Directions Further research is needed to gather high-quality benchmark data for partial diacritization, to enable more traditional and direct performance metrics alongside the indicators we propose. One promising direction is evaluating the
method on other languages which utilize diacritical marks in their orthographies