Script Normalization for Unconventional Writing of Under-Resourced Languages in Bilingual Communities
Year: 2023

Abstract:
The wide accessibility of social media has provided linguistically under-represented communities with an extraordinary opportunity to create content in their native languages. This, however, comes with certain challenges in script normalization, particularly where the speakers of a language in a bilingual community rely on another script or orthography to write their native language. This paper addresses the problem of script normalization for several such languages that are mainly written in a Perso-Arabic script. Using synthetic data with various levels of noise and a transformerbased model, we demonstrate that the problem can be effectively remediated. We conduct a small-scale evaluation of real data as well. Our experiments indicate that script normalization is also beneficial to improve the performance of downstream tasks such as machine translation and language identification.1

heading: 1 Introduction
text: With the increasing accessibility of the internet around the globe, from nomad settlements to megacities, we are living in a critical era that gives voice to different language communities to be represented online. Text as the essential material of many applications in natural language processing (NLP) may not be consistently created by under-represented communities due to the lack of a writing system and/or tradition, a widely-used orthography, and reasons related to regional and geopolitical conditions. Moreover, speakers of a language may opt for another script or prefer to use the script of an administratively dominant language other than the conventional one used in their native language (Gathercole and Thomas, 2009). This can be due to illiteracy in the native or home language, diglossia, or simply, the lack of adequate support for specific languages and scripts re-
1The data and code are publicly available at https:// github.com/sinaahmadi/ScriptNormalization.
sulting in remarkable negative effects on writing (Oriyama, 2011; Eisenchlas et al., 2013). Despite the fascinating progress in language technology in recent years, there are many fundamental tasks in NLP that are usually deemed trivial yet are ubiquitous, needed, and still unsolved in low-resource settings. One of these tasks is to deal with noisy text which affects the quality and performance of various applications such as text mining (Dey and Haque, 2009) and machine translation (Sperber et al., 2017). In practice, some steps are generally taken to deal with inconsistencies in the text, such as unifying character encoding and rectifying orthographic rules – a task broadly referred to as data cleaning or text preprocessing (Chu et al., 2016). However, unconventional and unsystematic writing such as using the script of a language towrite in another language presents challenges beyond traditional data cleaning scenarios. In this vein, we address the task of script normalization which aims to normalize a text writ-
14466
Persian ەکزرب رس هگایپ تیب قاس شل و ژیرد نمت ەکەزرەبرەس ەگایپ تیب قاس شەل و ژێرد نەمەت
ten in a script or orthography other than the one that is widely used in the language as the conventional one. For instance, writing Kashmiri or Sorani Kurdish in a script like that of Urdu or Persian, rather than in their own conventional scripts. Although this task has much in common with data cleaning, transliteration (Ahmadi, 2019), text normalization as defined by Sproat and Jaitly (2016) and spelling error correction (Hládek et al., 2020), it is particular in a few ways: unconventional writing does not necessarily comply with orthographic rules of the text’s language; when writing borrowed or common words from the dominant language, there is an influence of the orthography of the donor language rather than writing in the recipient language’s script or orthography; phonemes and graphemes may not be represented according to any specific rule, as in writing /S/ as ‘ch’, ‘sh’ or ‘$’ in Arabizi (Al-Badrashiny et al., 2014) following an erratic or obscure pattern among speakers leading to a huge amount of noisy material. A few examples of unconventional writing in Kurdish, Kashmiri and Gilaki are shown in Table 1.
In this paper, we focus on the script normalization of a few under-resourced languages that use variants of the Perso-Arabic script, as schematized in Figure 1, and are spoken as minority languages in bilingual communities with a dominant language using a similar but different script. These languages are Azeri Turkish (AZB), Mazanderani (MZN), Gilaki (GLK), Sorani Kurdish (CKB), Kurmanji Kurdish (KMR), Gorani (HAC), Kashmiri (KAS) and Sindhi (SND). Although these languages have their own customized scripts with more or less defined orthographies in their communities, they are oftentimeswritten in the script of the dominant language, notably Persian (FAS), Arabic (ARB)
and Urdu (URD) scripts. Furthermore, these languages have been lexically and, to a lesser extent, typologically influenced by the administratively dominant languages. Akin to many other multilingual and pluricultural societies, speakers of these languages have faced language conflict and linguistic discrimination in different educational, political, cultural, and communicative domains, and struggle with ethnolinguistic vitality (Mohan, 1989; Shah, 1997; Bird, 2020; Sheyholislami, 2022). Appendix A presents a summary of the languages we study.
Contributions In this work, we aim to: 1. shed light on script normalization for under-
resourced languages with very limited progress in language technology to facilitate the identification and collection of relevant data in the future, 2. leverage synthetic data for script normalization by mapping scripts based on rules and sequence alignment, and 3. cast script normalization as a translation task where noisy text is “translated” into normalized one using synthetic data generated.
We demonstrate that imposing different levels of noise on the synthetic data is beneficial to train more robust transformer-based models to normalize scripts and also, improve the performance of downstream tasks such as machine translation and language identification of unconventional writing.

heading: 2 Related Work
text: Although some aspects of script normalization have been previously addressed in related NLP tasks, its definition is a rather subjective matter, where a set of intentional or unintentional anoma-
lies are “normalized”. Therefore, script normalization overlaps considerably with more well-defined tasks such as spelling correction, lexical normalization (van der Goot et al., 2021) where an utterance is transformed into its standard form, pattern recognition (Schenk et al., 2009; Maddouri et al., 2000), language identification and standardization (Partanen et al., 2019; Ahmadi, 2020c). Script and text normalization have been proven beneficial in various downstream applications such as dependency parsing (Zhang et al., 2013), sentiment analysis (Mandal and Nanmaran, 2018) and named-entity recognition (Baldwin and Li, 2015). Although in some contexts normalization has been used to refer to basic tasks such as stemming and lemmatization, as in (Toman et al., 2006), those are not within the scope of this paper.
Text Normalization One of the most related tasks to script normalization is text normalization which broadly deals with alternative spellings, typos, abbreviations and non-canonical language and is of importance to text-to-speech systems and for handling micro-blogging data such as Tweets (Sproat and Jaitly, 2016). To this end, a wide range of techniques have been proposed using rules based on annotated corpora (Sigurðardóttir et al., 2021) or linguistic information (Xia et al., 2006), edit operations and recurrent neural networks (Chrupała, 2014), machine translation (Graliński et al., 2006), supervised learning (Yang and Eisenstein, 2013), encoder-decoders (Lusetti et al., 2018) and more recently, transformers (Zhang et al., 2019; Tan et al., 2020; Bucur et al., 2021). MoNoise (van der Goot and van Noord, 2017) is a prominent approach to text normalization where the problem is framed as a domain adaptation one and various steps are taken to generate and rank normalized candidates using spell checkers, word embeddings, dictionaries and n-grams features.
Perso-Arabic Script Normalization As one of the important scripts adopted by languages spoken by over 600 million speakers (Doctor et al., 2022), the Perso-Arabic scripts are prevalent on the Web nowadays. Although script normalization in general and addressing ambiguities of writing systems, in particular, have been previously addressed in the related tasks for such languages, such as Arabic (Ayedh et al., 2016; Shaalan et al., 2019), Kashmiri (Lone et al., 2022a), Kurdish (Ahmadi, 2019) and Sindhi (Jamro, 2017), normaliz-
ing Perso-Arabic scripts has not received much attention, let alone for noisy data originated from unconventional writing in bilingual communities. In a recent study, Doctor et al. (2022) address the normalization of Perso-Arabic script for a number of languages, namely Urdu, Punjabi, Sindhi, Kashmiri, Sorani Kurdish, Uyghur and Azeri Turkish. Inspired by Johny et al. (2021)’s approach to using finite-state transducers (FSTs) to normalize Brahmic scripts, Gutkin et al. (2022) implement FSTs for Perso-Arabic scripts for Unicode normalization, visual normalization and reading normalization by focusing on normalization and unification of varieties based on regional orthographies rather than that of specific dominant scripts.
Low-resource Setup Most under-resourced languages that require script normalization face the predicament of data paucity. On the other hand, data annotation is a tedious task that may not be always feasible for all languages. To tackle these, Dekker and van der Goot (2020) create synthetic data in which canonical words are replaced with non-canonical ones. Lusito et al. (2022) use a transformer-based model and employ modern data augmentation techniques for the endangered language of Ligurian; to deal with the scarcity of data, “back-normalization” is proposed where normalized text is transformed to a noisy one, analogous to back-translation (Sennrich et al., 2016). Similarly, many other studies rely on the synthetic generation of noisy data for tasks such as grammatical error correction (Foster and Andersen, 2009), creating noise-resistant word embedding (Doval et al., 2019;Malykh et al., 2018) andmachine translation (Bogoychev and Sennrich, 2019). In comparison to the previous work, our work focuses on text anomalies caused by the usage of a different script in bilingual communities. In addition, we propose modeling the problem as a machine translation and generating synthetic data by script mapping. To the best of our knowledge, our approach to this problem has not been previously explored for the selected languages.

heading: 3 Methodology
text: This section presents our methodology to collect data, create script mapping to generate synthetic data and implement a transformer-based model. Source and dominant in this context respectively refer to the language of the original text and that of the dominant one used unconventionally.

heading: 3.1 Data Collection
text: As the first step, we collect data written in the conventional script of the selected languages. To that end, we create corpora based on Wikipedia dumps.2 Since Wikipedia is not available for Gorani, we use Ahmadi (2020b)’s corpus for Gorani. Unlike the Latin script of Kurmanji for which there are corpora and Wikipedia, such as Pewan (Esmaili et al., 2013), there is no corpus for Kurmanji written in its Perso-Arabic script. Instead of relying on unreliable transliteration tools to convert the existing Kurmanji data, we crawl data from mainly news websites in the Iraqi Kurdistan for Kurmanji using the Perso-Arabic script.3 It is worth mentioning that for Sorani Kurdish we use a large existing corpus (Ahmadi and Masoud, 2020) instead of the (smaller) Wikipedia dump. We clean raw text by removing hyperlinks, email addresses, dates, non-relevant symbols and zero-width non-joiner (ZWNJ), if not systematically used in the script. Different types of numerals, namely Eastern Arabic <٠١٢٣٤٥٦٧٨٩>, Farsi <۰۱۲۳۴۵۶۷۸۹> and Hindu-Arabic <0123456789>, are unified with the later ones for consistency. We also deal with script switching in some Wikipedia articles, particularly in Sindhi and Kashmiri, using regular expressions to only keep the Perso-Arabic data. Following this, we extract vocabularies from the corpora based on a frequency list; depending on the size and quality of the data, we select words appearing with a minimum frequency in the range of 3 to 10. In addition to the vocabulary extracted from corpora, we also collect word lists and bilingual dictionaries in the source and target languages from other sources online. We consulted Wiktionary4 for Azeri Turkish, Kashmiri, Mazanderani, Sindhi and Sorani without finding any such resources for the other languages. Additionally, the lexicographical data provided by Ahmadi et al. (2019) were used for Sorani.

heading: 3.2 Script Mapping
text: To simulate the process that leads to noisy data, we create script mappings that map characters in the conventional script of the source language to that of the dominant language. To do so, we define mapping rules between the scripts based on
2Dumps of December 1, 2022. 3This corpus will be released along with the other data. 4https://www.wiktionary.org
the orthographies of the languages, as in the compound characters <ێئ> in Kurdish (composed of <ئ> (U+0626) and <ێ> (U+06CE)) that appear so only at the beginning of a word and this can be mapped to either <ا> (U+0627) or the same character but with the diacritic Kasrah as <ِا>. In addition, we take the closest candidates in the other script into account according to Unicode normalization as in <ک> (U+06A9) and <ك> (U+0643), and visual normalization, i.e. resemblance of the graphemes as in <ڎ> (U+068E) and <ذ> (U+0630). Table 2 shows a few mapping rules.
Using the rule-based script mappings, we also determine words in the word lists and bilingual dictionaries that are potential translations and written similarly in the two scripts. We also consider removing diacritics, also known as Harakat, as they are not always included in writing. The following words are collected this way: ‘جنرب’ (‘rice’) in Kurmanji and Persian, ‘یدیوس’ (‘Swedish’) written with <ی> (U+06CC) in Sorani and ‘يديوس’ written with <ي> (U+064A) in Arabic, ‘یکیٖرماَ’ (‘American’) in Kashmiri and ‘یکیرما’ in Urdu and, ‘جرۆب’ (‘tower’) in Azeri Turkish and ‘جرب’ in Persian by removing <ۆ> (U+06C6). As such, we refer to the set of words collected as spelling pairs.

heading: 3.3 Character-alignment Matrix
text: Although script mapping based on rules and modifications is effective, especially to retrieve common words or words borrowed by the two languages, it may lead to potentially false friends or incorrect spelling pairs as well. Hence, to capture information based on the spelling pairs, we rely on the character alignment of words. To this end, we employ Needleman and Wunsch (1970)’s algorithm for sequence alignment that maximizes the number of matches between sequences, i.e. words,
a and b with respect to the length of the two sequences. Therefore, we define the alignment matrix D for each spelling pair by setting elements i in a and j in b according to the following:
Di,j = max    Di−1,j−1 + daibj
Di−1,j + w Di,j−1 + w
where Di,j is the score of character i in the sequence a and character j in the sequence b, daibj denotes the similarity score of the characters at i and j (1 if similar and -1 otherwise) and, w refers to the gap penalty which is set to -1. A gap penalty is a penalizing score for non-matching characters and is shown by ‘-’ in our implementation. The matrix is initialized with D0,0 = 0. This algorithm is beneficial for our task as it considers the two sequences globally and allows back-tracing, hence useful to provide sequence matches. The following example shows the alignment of ‘مانتەیڤ’ (‘Vietnam’) in Sorani with the same word ‘مانتیو’, in Persian using this algorithm:
مانتەیڤ م ا ن ت ە ی ڤ مانتیو م ا ن ت - ی و
Finally, we merge all the alignment matrices, i.e. Ds, and create a character-alignment matrix for each pair of source and dominant languages. This matrix is normalized to have a unit norm. Furthermore, the mappings based on the rules of script mapping described in the previous subsection are appended to the matrix with a probability of 1. Alignments with a score < 0.1 are removed from the matrix due to the low probability of replacement. Depending on the score, a character can be aligned to more than one character in the dominant script.

heading: 3.4 Synthetic Data Generation
text: Given that there is a limited amount of data available for the selected languages, and that identifying languages in noisy setups is a challenging task, we have to rely on synthetic data generation. To that end, we first extract sentences of minimal and maximal lengths of 5 and 20 tokens (space-delimited) from the corpora described in §3.1. Then, we replace characters in the extracted sentences (clean data) with those in the characteralignment matrix. The replacement is done randomly with uniform sampling from the character alignment matrices to increase diversity. Depend-
ing on the number of replacements, we also consider specific percentages of noise generation at 20%, 40%, 60%, 80% and 100%. We create a last dataset by merging all datasets with all levels of noise; this creates more noisy instances given the randomness of noise generation. The parallel data are then tokenized using regular expressions. The number of parallel sentences and words per noise scale is provided in Appendix C.

heading: 3.5 Implementation and Evaluation
text: We use JoeyNMT (Kreutzer et al., 2019) to train transformer encoder-decoder models at the character level based on the degree of noise and language pairs. Using this model, our objective is to encode noisy data, i.e. synthetically noisy sentences, and decode normalized ones, i.e. the clean sentences in the collected datasets. We report the performance of the models by BLEU score (Papineni et al., 2002) and character n-gram F-score (chrF Popović, 2015), both calculated based on SacreBLEU (Post, 2018), along with sequence-level accuracy (Seq. Acc.), i.e. number of correct tokens in the hypothesis appearing in the same position as the reference divided by all tokens. Hyperparameter details are outlined in Appendix B. We evaluate the performance of the trained models based on the noise scale. As a naive baseline system, we calculate the evaluation metrics for the parallel data without applying any normalization technique. Finally, we evaluate the effectiveness of our models in two downstream tasks: language identification and machine translation.

heading: 4 Results and Analysis

heading: 4.1 Script Normalization
text: As the first set of experiments, we evaluate the performance of the script normalization models on synthetic noisy data at all levels of noise. In Appendix, Table D.3 provides the results of all the models and compares them with the baseline, and examples of normalization by our models are provided in Table D.1. Furthermore, the performance of a few selected models is presented in Figure 2. Although our models perform competitively in comparison to the baseline, performance is not identical across the datasets. By increasing the noise level from 20% to 100% making the source data harder to be normalized, a gradual decrease in performance is expected, and we indeed observe this for both the baseline and our model, but for
most datasets (7 out of 12) the degradation for the naive baseline is more rapid and pronounced. Our model does seem to handle various levels of noise: in Sindhi, for instance, we get 75.1 BLEU score vs. 19.7 of the baseline (see bottom right SND100URD→SND in Figure 2).
For five datasets, namely, AZBFAS→AZB, GLKFAS→GLK, MZNFAS→MZN, KASURD→KAS and HACCKB→HAC, the naive baseline outperforms our models. We believe that this is explained by the level of similarity of the source and dominant scripts, which in turn determines the difficulty of script normalization. To quantify our assumption, we define RA:B as the script ratio of scripts A and B, both as two sets of characters, as follows:
RA:B = A ∩B A ∪B × A ⋒B A ∩B
where A ⋒ B refers to the intersection of characters in scripts A and B which are mapped in the rule-based script mapping (see §3.2) without any other alternative in the other script while A ∩B is the intersection of A and B regardless of the mapping. Intuitively speaking, the script ratio of two identical scripts should be closer to 1 while more different scripts with various mappings between characters should get a lower value. Table A.1 provides the script ratios. Figure 3 shows the BLEU score (left y-axis) of the baseline and our model to normalize the datasets containing 100% noise, e.g. GLK100FAS →GLK along with the script ratio for each language (right y-axis). This indicates that the normalization model (model100) performs better where the script ratio is relatively low (<0.6, i.e. the two scripts are not that similar). On the other hand, the baseline performs better for higher script ratios, because in general the two scripts are very similar and hence there is less “noise”. We leave for future work an exploration as to why our transformer models fail to even simply learn to copy their input to perform at least on par with the baseline.
Real Data Evaluation Given the scarcity of real data, we resorted to generating synthetic data for training models and consequently evaluating them. However, working with real data is also crucial to evaluate the effectiveness of our approach. As such, we collected 100 sentences from social media in Sorani Kurdish written in unconventional scripts of Persian and Arabic. These sentences are then manually normalized by native and expert speakers based on the standard orthography of Kurdish.
The results of the small-scale evaluation on the real data are provided in Table 3. In these datasets, calculating BLEU scores of the source sentences (noisy) with respect to the reference ones (clean) for CKBFAS→CKB and CKBARB→CKB gets to 1.2 and 0.4 points, respectively. Once the source sentences are normalized using model100, the corresponding BLEU scores increase to 20.1 for CKBFAS→CKB and 12.7 for CKBARB→CKB. We selected this model as it has been trained on the most diverse training set. We believe that such a boost in BLEU scores indicates the robustness of our models to effectively normalize unconventional writing.

heading: 4.2 Language Identification
text: Language identification is the task of detecting the language of a text, usually a sentence. It is modeled as a probabilistic classification problem. As the first downstream task, we carry out a few experiments on language identification in three setups:
1. CLEAN: identifying languages without injecting any noise in the datasets, i.e. the target sentences in the parallel data. This is equivalent to 0% of noise in the data. 2. NOISY: identifying languages with noisy data at various levels, starting from 20% of noise and gradually increasing 20% up to 100%. We combine all data with all levels of noise in a separate dataset referred to as ALL. 3. MERGED: merging CLEANwith ALL dataset, i.e. with all noisy data.
We use the Tatoeba sentence dataset5 for data in Persian, Urdu and Arabic, with additional data for Urdu from the TED corpus on Opus (Tiedemann,
5https://tatoeba.org
2012). To tackle data imbalance, we downsampled all the datasets to only include 6000 sentences for each language6. In the MERGED setup where there are 12,000 sentences per language (half noisy, half clean), additional sentences (clean) in Persian, Urdu and Arabic are added to avoid data imbalance. Finally, we then split datasets into train and test sets with an 80-20% split. To train supervised language identification models, we use fastText (Bojanowski et al., 2017) with subword features with minimum and maximum character ngram sizes of 2 and 4, word vectors of size 16 and hierarchical softmax as the loss function. Table 4 presents the results of the performance of our models in comparison to fastText’s language identification model trained on data from Wikipedia, Tatoeba and SETimes on 176 languages,7 including, Persian, Arabic, Urdu, Sindhi, Sorani and Mazanderani. Although Azeri Turkish is supported, it is not clear which script it is trained on in fasText. The results are reported based on precision, recall and F1 score of the first and second detection of the models, respectively denoted by ‘@1’ and ‘@2’. SinceGorani andGilaki are not among the Fairseq-supported languages, we focus our analysis on the top-two predictions (@2) to en-
6Kashmiri had only 4700 instances. 7As of January 2023.
sure fairness against the baseline. Our models outperform fastText in noisy conditions, regardless of the level of noise. The baseline performance degrades faster as the level of noise increases. Our models perform less effectively on clean data but recall that they are trained on a substantially smaller amount of data.8
Figure 4 illustrates the results of classification in our MERGED model where the detected languages (x-axis) are compared with the references (y-axis). Although the model performs well in detecting Arabic, Persian, Sindhi, Urdu and Kashmiri, the other languages are often confused. The misclassified languages can be categorized into two groups of [Azeri, Gilaki, Mazanderani] and [Sorani, Kurmanji, Gorani]. This can be explained by the similarity of scripts. Surprisingly, Sindhi and Kashmiri represent a less salient overlap in classification.

heading: 4.3 Neural Machine Translation
text: Furthermore, we evaluate the performance of the script normalization models in neural machine translation (NMT). To do so, we use the devtest data of the FLORES 200 dataset (Costa-jussà et al., 2022) that contains 1012 parallel sentences for 204 languages including Kashmiri, Sorani and Sindhi in their Perso-Arabic scripts along with English. With English as the target language and the other available languages as sources, we carry out the evaluation in the following three setups:
8We conduct a similar study on a wider range of languages in (Ahmadi et al., 2023).
1. CLEAN: translating the devtest of the source languages as it is in the FLORES dataset (with no noise), e.g. CKB→ENG for translating Sorani (CKB) to English (FAS). This is a skyline setting (best possible). 2. NOISY: translating the devtest by synthetically applying a certain amount of noise. For this setting, we specify the maximum amount of “interference” with the dominant language’s script (100% following the previous notation). The noise generation is random similar to language identification (§3.4). This is the baseline setting (worst possible, no normalization). 3. NORMALIZED: we normalize the data from the NOISY setting with our text normalization models (§4.1), and then translate the normalized text to English.
We use Facebook’s No Language Left Behind (NLLB) model on HuggingFace9 which is trained on parallel multilingual data from a variety of sources to translate into English and evaluate using SacreBLEU (Post, 2018). Table D.2 presents results of the translation quality of the NLLB model in our intended setups, with a sample in Table 5. Except for KASURD→ENG, the translation quality of unnormalized data (S) deteriorates as noise levels increase while it remains steady when using normalization models as a preprocessing step (H). In Sorani, for example, even with a 100% of noise (CKB100ARB), our normalization approach (H) recovers almost 20 BLEU points out of the 26 points that were lost due to the non-conventional setting. The importance of a trustworthy text normalization system is clear in the case of Kashmiri, where our normalization model fails to reduce noise, resulting in subpar performance. From a qualitative point of view, the noisy input affects the quality of translations depending on the type of noise. Although the NLLB model shows resilience to certain types of errors, particularly where a character is substituted by an incorrect one without an impact on word boundaries, i.e. no spaces being added or removed, noisy characters that can possibly affect tokenization lead to incorrect translations. For instance in Table D.4, the translation of noisy sentences in Sindhi is less different from that of the reference, while the Sorani noisy sentence is translated terribly differently and
9The nllb-200-distilled-600M variant.
incorrectly. Needless to say, the meaning of some words varies with incorrect characters.

heading: 5 Conclusion and Discussion
text: This paper discusses the challenge of script normalization of unconventional writing of languages that are spoken in bilingual communities. Under the influence of the dominant language of such communities, speakers tend to use scripts or orthographies of the dominant language rather than the one that is conventionally used in their native language. This leads to noisy data that hinder NLP applications and reduce data availability. Framing the problem as a machine translation one where noisy data is ‘translated’ into clean one, we address the task of script normalization for the Perso-Arabic scripts of several languages, namely Azeri Turkish, Mazanderani, Gilaki, Sorani and Kurmanji Kurdish, Gorani, Kashmiri and Sindhi, with dominant languages being Arabic, Persian and Urdu. Given that these languages are lessresourced, we rely on synthetic data to create parallel datasets by injecting noise based on script mapping. We then train transformer-based encoderdecoders and show that the models can tackle the problem effectively, even for real data, but differently based on the level of noise. We demonstrate that script normalization can be effectively addressed, particularly where there are many dissimilarities between the source and dominant languages. It can also alleviate the impact of noisy data on downstream tasks, namely language identification and machine translation. In
addition to script normalization that can help retrieve texts for the selected under-resourced languages, the trainedmodels alongwith the collected data, with or without noise, can pave the way for further developments for those languages.
Limitations One of the limitations of the current study is the lack of annotated data for all languages. This is also the case of machine translation for which data could only be found for Kashmiri, Sorani and Sindhi, while other languages do not have much parallel data yet. On the other hand, the notion of noisy data is limited to the replacement of the missing characters in a script when compared to another one, i.e. that of the dominant language. As an ablation study, injecting other types of noise, beyond those discussed in this paper, may improve the performance of the models to tackle not only script normalization but several related tasks such as spelling error correction and may also increase the robustness of the models for morphologically rich languages or languages with versatile word boundaries using ZWNJ. Although we did our best to filter out code-switched data in the corpora, our datasets may contain data in other languages (in Perso-Arabic scripts). In future work, we would like to apply our approach to other scripts and languages in bilingual communities. We also suggest evaluating the impact of script normalization on more downstream tasks, especially transliteration and tokenization.
Ethics Statement
All corpora and datasets used in this study are publicly available, ensuring compliance with data privacy regulations. Although we did our best to remove any personally identifiable information and preserve the privacy and anonymity of individuals, it is possible that some of the selected corpora contain sensible or offensive information. Filtering such content is challenging given that all the targeted languages are low-resourced and lack proper NLP tools for this purpose. We believe that it is unlikely that the normalization models cause benefits or harm to individuals. Regarding the annotation of the data (§4.1), annotators were fairly compensated for their time and effort. By upholding these ethical principles, we aimed to conduct the study in a responsible and conscientious manner.

heading: Acknowledgments
text: This work is supported by the National Science Foundation under DLI-DEL award BCS-2109578. The authors are also grateful to the anonymous reviewers, as well as the Office of Research Computing at GMU, where all computational experiments were conducted.

heading: A Selected Languages
text: In this work, we focus on script normalization for a few languages that are spoken in bilingual communities and use a Perso-Arabic script. Even though the Perso-Arabic script is not limited to our selected languages, we did not include languages that are spoken in countries where the administratively dominant language uses another script, such as Uyghur and Malay, or those that have historically used a Perso-Arabic script that is now obsolete, like Dogri, Turkish and Tajik Persian. This said, there are other languages that would fit into our study but due to lack of data could not be included, such as Luri (LDD), Balochi, Shina, or Burushaski.
Azeri Turkish also known as Azerbaijani or Azari, is a Turkic language mainly spoken in Azerbaijan, Iranian Azerbaijan and broadly the Caucasus by 20 million speakers (Rezaei et al., 2017). The two varieties of Azeri Turkish, Northern Azeri Turkish and Southern Azeri Turkish respectively use the Latin and the Perso-Arabic scripts. In this study, we focus on the latter variety. The PersoArabic script of Azeri Turkish is similar to that of Persian, with additional graphemes such as <ؽ> (U+063D) and <ۇ> (U+06C7). Mazanderani also known as Mazandarani or Tabari (Borjian and Borjian, 2023), is an IndoAryan language spoken in regions adjoining the Caspian Sea in Iran, chiefly in Mazandaran Province, by 2.5-3 million speakers (Mirhosseini, 2015, p. 157). The Perso-Arabic script adopted for Mazandarani is almost identical to the one used for Persian, with the additional diacritic <ˇ> (U+02C7).
Gilaki is an Indo-Aryan language, similar to Mazanderani spoken in regions adjoining the Caspian Sea in Iran by over 2 million speakers (Rastorgueva et al., 2012; Khoshrouz, 2021). Although Mazanderani and Gilaki have been historically written using the Persian script (Borjian, 2008) without a recognized orthography, there have been recently many efforts, particularly on Wikipedia10, to adopt a Latin-based script. This said, the Latin script did not survive and the PersoArabic scripts are used for both Gilaki and Mazanderani (Sedighi, 2023, p. 20). The Perso-Arabic script adopted for Gilaki has graphemes in addition
10https://www.wikipedia.org
to those in Persian, such as <ۋ> (U+06CB) and <ۊ> (U+06CA).
Kurdish is an Indo-Aryan language spoken by over 25 million speakers in Iraq, Iran, Turkey and Syria, in varieties classified as Northern Kurdish or Kurmanji, Central Kurdish or Sorani, and Southern Kurdish (Ahmadi andMasoud, 2020). Kurdish has historically employed various scripts for writing, such as Cyrillic, Armenian, Latin and PersoArabic. Among these, the two latter scripts are still widely used. While the Latin script is more popular for writing Kurmanji spoken in Turkey, speakers of other varieties prefer the Perso-Arabic script of Kurdish, mainly due to the widespread usage of those scripts by the regional administrations. This is particularly the case of Sorani and Kurmanji spoken in Iraq and to some extent, in Syria. The Kurmanji variant spoken in Kurdistan of Iraq is called Behdini (also spelled Badini). Kurdish is a phonemic alphabet which is known by its distinct characters such as <ۆ> (U+06C6) and <ێ> (U+06CE), and absence of varying phonetically-related graphemes of Arabic such as <ض> (U+0636) and <ظ> (U+0638) even for loanwords (Ahmadi, 2020a). It is worth noting that the Perso-Arabic script that is used for both Sorani and Kurmanji follow the same orthographies. However, this is less established for Kurmanji given the popularity of the Latin script for that dialect. Throughout the paper, we consider Kurmanji and Sorani differently due to the considerable lexical and morpho-syntactic differences. It is worth noting that Gorani speakers oftentimes use Sorani script without the additional characters of Gorani making Sorani a dominant language.
Gorani also known as Hawrami, is an IndoAryan language spoken by 300,000 speakers in the parts of the Iranian of Iraqi Kurdistan (Ahmadi, 2020b). Due to the mutual linguistic and cultural influences of Sorani and Gorani, Gorani speakers rely on the Perso-Arabic script and the orthography used for Kurdish. However, there are a few additional graphemes that are unique to Gorani and cannot be found in Kurdish, such as <ڎ> (U+068E) and <ۋ> (U+06CB). Kashmiri also known as Koshur, is an IndoAryan language spoken by over 7 million speakers in the disputed territories administered by three countries: India, Pakistan, and China (Lone et al.,
2022b). Although Kashmiri’s Perso-Arabic script relies much on that of Urdu, the extensive use of diacritics and distinct characters such as <ۄ> (U+06C4) and <ؠ> (U+0620) make the adopted script discernible. Moreover, Kashmiri extensively uses diacritics to specify vowels, while diacritization is less frequent in everyday writing of the other languages, except for disambiguation.
Sindhi is an Indo-Aryan language primarily spoken in Pakistan and parts of India by over 20 million speakers (David et al., 2017). Although influenced by Urdu, Sindhi uses a more diverse set of letters (62 vs. 40 in Urdu) (Doctor et al., 2022). It can be distinguished by unique letters such as <ڏ> (U+068F) and <ڻ> (U+06BB).
Similar to Persian and Urdu, Azeri Turkish, Mazanderani and Gilaki use the zero-width nonjoiner (ZWNJ, U+200C) character (Lockwood, 2011) which creates lexical variations and adds to the complexity of downstream tasks like tokenization (Ghayoomi andMomtazi, 2009; Rehman et al., 2011; Doostmohammadi et al., 2020). Furthermore, the usage of some graphemes such as <ض> (U+0636) and <ظ> (U+0638) in the adopted scripts is mainly limited to loanwords from Arabic and Persian, unless conventionally used to represent a specific phoneme in the language. Moreover, the final glyph <ھ> (U+06BE) is used in all languages except Persian, Mazanderani, Azeri Turkish and Gilaki. While the Perso-Arabic script of Kurdish, Gorani and Kashmiri are alphabets, the other selected languages are impure Abjads. Similarly, all the selected languages use Naskh style for writing, unlike Urdu that uses Nastaliq.
A summary of various aspects of the selected languages is provided in Table B.1. We also provide the script ratio (as defined in §4.1) of the script of the selected languages and that of the dominant one in Table A.1.

heading: B Models Details
text: We train the normalization models in different configuration of hyper-parameters as follows: • Training:
– level: character – maximum number of characters: 100 – character minimal frequency: 5 – optimizer: adam – learning rate: 0.001 – learning rate min: 0.0002 – patience: 5 – number of layers: 6 – number of heads: 8 – embedding dimension: 128 – hidden size: 128 – position-wise feed-forward layer size: 512
• Testing: – number of best prediction: 1 – beam size: 4 – beam alpha: 1.0 – max output length: 100 – batch size: 10
In addition to these common hyper-parameters, we set different hyper-parameters depending on the size of the datasets. The approximate overall number of pairs in all noisy datasets are provided in parentheses.
• (<10k pairs) KASURD, HACCKB epochs: 100 / dropout: 0.2 / batch size: 64 • (<50k pairs) KMRARB, HACARB, KMRFAS, HACFAS, GLKFAS, MZNFAS epochs: 80 / dropout: 0.2 / batch size: 64 • (<600k pairs) SNDURD, AZBFAS epochs: 40 / dropout: 0.1 / batch size: 64 • (<6.6M pairs) CKBARB, CKBFAS epochs: 6 / dropout: 0.1 / batch size: 10
For each configuration (72 in total), the training was carried out on a cluster using one GPU per configuration with 32GB memory.

heading: C Datasets
text: The number of words and sentence pairs in the synthetic datasets generated for the selected languages is provided in Table C.1.

heading: D Experiments Results and Examples
text: This section presents examples of script normalization and machine translation. Given the source sentence, the model trained on all noisy datasets, i.e. All, generates a hypothesis which is then translated. The selected sentences are taken from the 100% noisy dataset. The source, hypothesis and reference sentences are specified as S, H and R, respectively.
(a) Script normalization of sentences containing 100% noise
(b) Script normalization of a Sorani sentence with varying noise
Language S/H/R Sentence
CKBARB→CKB
S نةكةد راك ناطسدروك يناكةوارب و رادطلاةسةد ةبذح يزد
H نەکەد راک ناتسدروک یناکەواڕب و رادتڵاەسەد ەبزح یژد
R نەکەد راک ناتسدروک یناکەوارب و رادتڵاەسەد ەبزح یژد
GLKFAS→GLK
S هنئب أرمه شوخ هراگن و هنؤخ زيزع هنش دمحلأک
H هنئب أرمه ˇشۊخ هراگن و هنؤخ ˇزيزع هنش دمحلأک
R هنئب أرمه ˇشۊخ هراگن ؤ هنؤخ ˇزيزع هنش دمحلأک
HACFAS→HAC
S یمیینب ناشیمان درم مین ب یناناوز یا نشیو یگای
H ێمێینب ناشێمان ەدرەم مین ەب یناناوز یئ ەنەشێو ێگای
R یٛمیەینب ناشیٛمان ەدرەم ەمین ەب ەیناناوز یئ ەنەشیٛو یٛگای
KASURD→KAS
S ےنوٗہ ہمار ےھچ دام زنس ےنوٗہ ہمار
H !نوٗہ ہمار ےھچ دام زنسٕ !نوٗہ ہمار
R !نوٗہ ہٕمار ےٚھچ دٕام زنٛسٕ !نوٗہ ہٕمار
KMRFAS→KMR
S ر هص هک يژ هن لوک و در هد هن هنيک و بر هک ژ لااو نلد
H رەسەک یژ ەن لوک و درەد ەن ەنیک و ێبرەک ژ لااڤ نێلد
R رەسەک یژ ەن لوک و درەد ەن ەنیک و برەک ژ لااڤ نێلد
SNDURD→SND S يہآ ولان وج بکرم کھ م يرتسمیک ای ایمیک ملع دئائلاک H يھآ ولان وج بڪرم ڪھ م يرٽسمیڪ ای ایمیڪ ملع ڊئائلاڪ
R يھآ ولان وج بڪرم ڪھ ۾ يرٽسمیڪ ای ایمیڪ ملع ڊئائلاڪ
Source نارولاپ ن هص هر ينارگ نيرطشاب ب ،ناک هرطص ها يراش
Reference ناروێڵاپ نەسەڕ ینارۆگ نیرتشاب ۆب ،ناکەرێتسەئ یراش
City of Stars, received nominations for best original song
20 نارولاپ ن هص هڕ ینارگ نیرطشاب ب ،ناک هرطص ها یراش
40 ناروێڵاپ نەسەڕ ینارۆگ نیرتشاب ۆب ،ناکەرتسەئ یراش
60 اشناروێڵاپ نەصەڕ ینارۆگ نیرتشاب ۆب ،ناکەرطصەئ یراش
80 ناروێڵاپ نەسەڕ یناڕۆگ نیرتشاب ۆب ،ناکەرێتسەئ یراش

heading: 100 روێڵاپ ن ەس ەڕ ینارۆگ نیرتشاب ۆب ،نا’ ەرێتسەئ یراش
text: All ناروێڵاپ نەسەڕ ینارۆگ نیرتشاب ۆب ،ناکەرێتسەئ یراش
Table D.1: Examples of script normalization of sentences in different languages containing 100% noise (a) and in Sorani Kurdish with various levels of noise (b) using a model trained on all noisy datasets
Language Noise % Test set BLEU chrF
KAS100URD→ENG
0 R 28.40 55.71
20 S 28.42 55.70H 11.60 36.15 40 S 27.86 55.22H 11.38 36.03 60 S 27.88 55.14H 11.38 36.05 80 S 27.94 55.23H 11.31 35.96
100 S 27.93 55.13H 11.39 35.97
CKB100ARB→ENG
0 R 30.81 55.96
20 S 21.54 46.63H 24.52 50.43 40 S 15.16 39.49H 21.76 47.49 60 S 8.88 31.05H 19.87 45.73 80 S 4.75 24.59H 24.11 50.06
100 S 4.72 24.33H 24.53 50.55
CKB100FAS →ENG
0 R 29.69 45.79
20 S 21.04 39.95H 23.27 38.07 40 S 17.61 29.14H 17.29 30.41 60 S 21.04 38.56H 26.31 44.04 80 S 9.98 25.43H 19.45 35.45
100 S 17.29 25.39H 26.27 46.7
SND100URD→ENG
0 R 30.81 55.96
20 S 23.15 48.29H 23.57 49.32 40 S 18.03 42.60H 21.84 47.57 60 S 12.42 35.53H 19.43 45.23 80 S 6.95 27.45H 23.39 49.51
100 S 7.09 27.59H 23.97 49.47
TableD.2: Evaluation of NLLBmodel for translation of reference data (R), noisy data (S) and normalized data (H) using normalization models trained on all levels of noise (All)
Language TestNoise% Sentence
RSND .وھجمس يک ٽور گنیئاڪسا رگناو ٽور گنڪئاھ يئڳاس RENG Think of the skiing route as of a similar hiking route.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:

heading: 3 A1. Did you describe the limitations of your work?
text: Section 5
A2. Did you discuss any potential risks of your work? Not applicable. Left blank.

heading: 3 A3. Do the abstract and introduction summarize the paper’s main claims?
text: Abstract section and Section 1 as an introduction
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? Left blank.

heading: 3 B1. Did you cite the creators of artifacts you used?
text: Section 3

heading: 3 B2. Did you discuss the license or terms for use and / or distribution of any artifacts?
text: Sections 2 and 3

heading: 3 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided
text: that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Left blank.

heading: 3 B4. Did you discuss the steps taken to check whether the data that was collected / used contains any
text: information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Section 3.1
B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? Not applicable. Left blank.
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Appendix
C 3 Did you run computational experiments? Section 3

heading: 3 C1. Did you report the number of parameters in the models used, the total computational budget
text: (e.g., GPU hours), and computing infrastructure used? Appendix B
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.

heading: 3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found
text: hyperparameter values? Appendix B

heading: 3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary
text: statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Appendix

heading: 3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did
text: you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Section 3 and appendix
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? No response.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? No response.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? No response.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? No response.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? No response.

heading: Future Work
text: In future work, we would like to apply our approach to other scripts and languages in bilingual communities. We also suggest evaluating the impact of script normalization on more downstream tasks, especially transliteration and tokenization