SlowGuess/ACL_2020_OCR huggingface dataset:
Sawsan Alqahtani  $^{1,2}$  and Ajay Mishra  $^{1}$  and Mona Diab  $^{2*}$
$^{1}$ AWS, Amazon AI
2The George Washington University
sawsa@amazon.com, misaja@amazon.com, mtdiab@gwu.edu
In many languages like Arabic, diacritics are used to specify pronunciations as well as meanings. Such diacritics are often omitted in written text, increasing the number of possible pronunciations and meanings for a word. This results in a more ambiguous text making computational processing on such text more difficult. Diacritic restoration is the task of restoring missing diacritics in the written text. Most state-of-the-art diacritic restoration models are built on character level information which helps generalize the model to unseen data, but presumably lose useful information at the word level. Thus, to compensate for this loss, we investigate the use of multi-task learning to jointly optimize diacritic restoration with related NLP problems namely word segmentation, part-of-speech tagging, and syntactic diacritization. We use Arabic as a case study since it has sufficient data resources for tasks that we consider in our joint modeling. Our joint models significantly outperform the baselines and are comparable to the state-of-the-art models that are more complex relying on morphological analyzers and/or a lot more data (e.g. dialectal data).
In contrast to English, some vowels in languages such as Arabic and Hebrew are not part of the alphabet and diacritics are used for vowel specification. In addition to pertaining vowels, diacritics can also represent other features such as case marking and phonological gemination in Arabic. Not including diacritics in the written text in such languages increases the number of possible meanings as well as pronunciations. Humans rely on the surrounding
context and their previous knowledge to infer the meanings and/or pronunciations of words. However, computational models, on the other hand, are inherently limited to deal with missing diacritics which pose a challenge for such models due to increased ambiguity.
Diacritic restoration (or diacritization) is the process of restoring these missing diacritics for every character in the written texts. It can specify pronunciation and can be viewed as a relaxed variant of word sense disambiguation. For example, the Arabic word  $\text{Elm}^2$  can mean "flag" or "knowledge", but the meaning as well as pronunciation is specified when the word is diacritized (  $\text{Ealamu}$  means "flag" while  $\text{Eilomo}$  means "knowledge"). As an illustrative example in English, if we omit the vowels in the word  $pn$ , the word can be read as  $pan$ ,  $pin$ ,  $pun$ , and  $pen$ , each of these variants have different pronunciations and meanings if it composes a valid word in the language.
The state-of-the-art diacritic restoration models reached a decent performance over the years using recurrent or convolutional neural networks in terms of accuracy (Zalmout and Habash, 2017; Alqahtani et al., 2019; Orife, 2018) and/or efficiency (Alqahtani et al., 2019; Orife, 2018); yet, there is still room for further improvements. Most of these models are built on character level information which help generalize the model to unseen data, but presumably lose some useful information at the word level. Since word level resources are insufficient to be relied upon for training diacritic restoration models, we integrate additional linguistic information that considers word morphology as well as word relationships within a sentence to partially compensate for this loss.
*The work was conducted while the author was with AWS, Amazon AI.
<sup>1</sup>Diacritics are marks that are added above, below, or in between the letters to compose a new letter or characterize the letter with a different sound (Wells, 2000).
<sup>2</sup>We use Buckwalter Transliteration encoding http://www.qamus.org/transliteration.htm.
8238
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8238-8247 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
In this paper, we improve the performance of diacritic restoration by building a multitask learning model (i.e. joint modeling). Multitask learning refers to models that learn more than one task at the same time, and has recently been shown to provide good solutions for a number of NLP tasks (Hashimoto et al., 2016; Kendall et al., 2018).
The use of a multitask learning approach provides an end-to-end solution, in contrast to generating the linguistic features for diacritic restoration as a preprocessing step. In addition, it alleviates the reliance on other computational and/or data resources to generate these features. Furthermore, the proposed model is flexible such that a task can be added or removed depending on the data availability. This makes the model adaptable to other languages and dialects.
We consider the following auxiliary tasks to boost the performance of diacritic restoration: word segmentation, part-of-speech (POS) tagging, and syntactic diacritization. We use Arabic as a case study for our approach since it has sufficient data resources for tasks that we consider in our joint modeling. $^3$
The contributions of this paper are twofold:
1. We investigate the benefits of automatically learning related tasks to boost the performance of diacritic restoration;
2. In doing so, we devise a state-of-the-art model for Arabic diacritic restoration as well as a framework for improving diacritic restoration in other languages that include diacritics.
2 Diacritization and Auxiliary Tasks
We formulate the problem of (full) diacritic restoration (DIAC) as follows: given a sequence of characters, we identify the diacritic corresponding to each character in that sequence from the following set of diacritics  $\{\mathrm{a},\mathrm{u},\mathrm{i},\mathrm{o},\mathrm{K},\mathrm{F},\mathrm{N},\sim ,\sim \mathrm{a},\sim \mathrm{u},$ $\sim \mathrm{i},\sim \mathrm{F},\sim \mathrm{K}$  and  $\sim \mathbf{N}\}$ . We additionally consider three auxiliary tasks: syntactic diacritization, part-of-speech tagging, and word segmentation. Two of which operate at the word level (syntactic diacritization and POS tagging) and the remaining tasks (diacritic restoration and word segmentation) operate at the character level. This helps diacritic restoration utilize information from both charac
ter and word level information, bridging the gap between the two levels.
Syntactic Diacritization (SYN): This refers to the task of retrieving diacritics related to the syntactic positions for each word in the sentence, which is a sub-task of full diacritic restoration. Arabic is a templatic language where words comprise roots and patterns in which patterns are typically reflective of diacritic distributions. Verb patterns are more or less predictable however nouns tend to be more complex. Arabic diacritics can be divided into lexical and inflectional (or syntactic) diacritics. Lexical diacritics change the meanings of words as well as their pronunciations and their distribution is bound by patterns/templates. In contrast, inflectional diacritics are related to the syntactic positions of words in the sentence and are added to the last letter of the main morphemes of words (word finally), changing their pronunciations.4 Inflectional diacritics are also affected by word's root (e.g. weak roots) and semantic or morphological properties (e.g. with the same grammatical case, masculine and feminine plurals take different diacritics).
Thus, the same word can be assigned a different syntactic diacritic reflecting syntactic case, i.e. depending on its relations to the remaining words in the sentence (e.g. subject or object). For example, the diacritized variants Ealama and Ealamu which both mean "flag" have the corresponding syntactic diacritics: a and u, respectively. That being said, the main trigger for accurate syntactic prediction is the relationships between words, capturing semantic and most importantly, syntactic information.
Because Arabic has a unique set of diacritics, this study formulates syntactic diacritization in the following way: each word in the input is tagged with a single diacritic representing its syntactic position in the sentence. The set of diacritics in syntactic diacritization is the same as the set of diacritics for full diacritic restoration. Other languages that include diacritics can include syntactic related diacritics but in a different manner and complexity
<sup>4</sup>Diacritics that are added due to passivization are also syntactic in nature but are not considered in our syntactic diacritization task. That said, they are still considered in the full diacritic restoration model.
5 Combinations of diacritics is possible but we combine valid possibilities together as one single unit in our model. For example, the diacritics  $\sim$  and a are combined to form an additional diacritic  $\sim \mathbf{a}$
3Other languages that include diacritics lack such resources; however, the same multitask learning framework can be applied if data resources become available.
compared to Arabic.
Word segmentation (SEG): This refers to the process of separating affixes from the main unit of the word. Word segmentation is commonly used as a preprocessing step for different NLP applications and its usefulness is apparent in morphologically rich languages. For example, the undiacritized word whm  $\rho_{\phi}$  might be diacritized as waham~a "and concerned", waham "illusion", where the first diacritized word consists of two segments "wa ham~a" while the second is composed of one word. Word segmentation can be formulated in the following way: each character in the input is tagged following IOB tagging scheme ( $B$ : beginning of a segment;  $I$ : inside a segment;  $O$ : out of the segment) (Diab et al., 2004).
Part-Of-Speech Tagging (POS): This refers to the task of determining the syntactic role of a word (i.e. part of speech) within a sentence. POS tags are highly correlated with diacritics (both syntactic and lexical): knowing one helps determine or reduce the possible choices of the other. For instance, the word ktb in the sentence ktb [someone] means "books" if we know it to be a noun whereas the word would be either katab "someone wrote" or kat~ab "made someone write" if it is known to be a verb.
POS tagging can be formulated in the following way: each word in the input is assigned a POS tag from the Universal Dependencies tagset (Taji et al., 2017).<sup>6</sup>
We built a diacritic restoration joint model and studied the extent to which sharing information is plausible to improve diacritic restoration performance. Our joint model is motivated by the recent success of the hierarchical modeling proposed in (Hashimoto et al., 2016) such that information learned from an auxiliary task is passed as input to the diacritic restoration related layers.[7]
3.1 Input Representation
Since our joint model may involve both character and word level based tasks, we began our investigation by asking the following question: how to integrate information between these two levels? Starting from the randomly initialized character embeddings as well as a pretrained set of embeddings for words, we follow two approaches (Figure 1 visually illustrates the two approaches with an example).
Figure 1: An example of embedding vectors for the word  $cat$  and its individual characters: c,a, and t. (i) A character-based representation for the word  $cat$  from its individual characters; (ii) A concatenation for the word embedding with each of its individual characters.
(1) Character Based Representation: We pass information learned by character level tasks into word level tasks by composing a word embedding from the word's characters. We first concatenate the individual embeddings of characters in that word, and then apply a Bidirectional Long Short Term Memory (BiLSTM) layer to generate denser vectors. This helps representing morphology and word composition into the model.
(2) Word-To-Character Representation: To pass information learned by word level tasks into character level tasks, we concatenate each word with each of its composed characters during each pass, similar to what is described in Watson et al. (2018)'s study. This helps distinguishing the individual characters based on the surrounding context, implicitly capturing additional semantic and syntactic information.
<sup>6</sup>Refer to https://universaldependencies.org/. This tagset is chosen because it includes essential POS tags in the language, and it is unified across different languages which makes it suitable to investigate more languages in the future.
<sup>7</sup>We also experimented with learning tasks sharing some levels and then diverging to specific layers for each tasks. However, this did not improve the performance compared to the diacritic restoration model when we don't consider any additional task.
We also evaluated the use of a feedforward layer and unidirectional Long Short Term Memory (LSTM) but a BiLSTM layer yielded better results.
(i) Input representation
(ii) Diacritic Restoration
Figure 2: The diacritic restoration joint model. All Char Embed entities refer to the same randomly initialized character embedding learned during the training process. Pretrained embeddings refer to fixed word embeddings obtained from fastText (Bojanowski et al., 2017). (i) shows the input representation for CharToWord and WordToChar embedding which is the same as in Figure 1. (ii) represents the diacritic restoration joint model; output labels from each task are concatenated with WordToChar embedding and optionally with segmentation hidden.
3.2 The Joint Model
For all architectures, the main component is BiLSTM (Hochreiter and Schmidhuber, 1997; Schuster and Paliwal, 1997), which preserves the temporal order of the sequence and has been shown to provide the state-of-the-art performance in terms of accuracy (Zalmout and Habash, 2017; Alqahtani et al., 2019). After representing characters through random initialization and representing words using pretrained embeddings obtained from fastText (Bojanowski et al., 2017), the learning process for each batch runs as follows:
1. We extract the two additional input representation described in Section 3.1;
2. We apply BiLSTM for each of the different tasks separately to obtain their corresponding outputs;
3. We pass all outputs from all tasks as well as WordToChar embedding vectors as input to the diacritic restoration model and obtain our diacritic outputs.
Figure 2 illustrates the diacritic restoration joint model. As can be seen, SYN as well as POS tagging are trained on top of CharToWord representation which is basically the concatenation of the pretrained embedding for each word with the character-based representations described in Figure 1. SEG is also trained separately on top of the
character embeddings. We pass the outputs of all these tasks along with WordToChar representation to train the BiLSTM diacritic restoration model. Omitting a task is rather easy, we just remove the related components for that task to yield the appropriate model. We optionally pass the last hidden layer for SEG along with the remaining input to the diacritic restoration model.<sup>9</sup>
4 Experimental Setups
Dataset: We use the Arabic Treebank (ATB) dataset: parts 1, 2, and 3 and follow the same data division as Diab et al. (2013). Table 1 illustrates the data statistics. For word based tasks, we segment each sentence into space tokenized words. For character based tasks, we, in addition, add the special boundary “<w>” between these words, and then each word is further segmented into its characters, similar to that in (Alqahtani et al., 2019). We pass each word through the model along with a specific number of previous and future words (+/- 10 words).
Parameter Settings: For all tasks, we use 250 hidden units in each direction (500 units in both directions combined) and 300 as embedding size. We use 3 hidden layers for tasks except in SEG in
<sup>9</sup>Passing the last hidden layer for POS tagging and/or SYN did not improve the performance; the pretrained embeddings are sufficient to capture important linguistic signals.
8241
Train	Test	Dev	OOV
502,938	63,168	63,126	7.3%
Table 1: Number of words and out of vocabulary (OOV) rate for Arabic. OOV rate indicates the percentage of undiacritized words in the test set that have not been observed during training.
which we use only one layer. We use Adam for learning optimization with a learning rate of 0.001. We use 20 for epoch size, 16 for batch size, 0.3 for hidden dropout, and 0.5 for embedding dropout. We initialize the embedding with a uniform distribution [-0.1,0.1] and the hidden layers with normal distribution. The loss scores for all considered tasks are combined and then normalized by the number of tasks in the model.
Evaluation metrics: We use accuracy for all tasks except diacritic restoration. For diacritic restoration, the two most typically used metrics are Word Error Rate (WER) and Diacritic Error Rate (DER), the percentages of incorrectly diacritized words and characters, respectively. In order to approximate errors in the syntactic diacritics, we use Last Diacritic Error Rate (LER), the percentage of words that have incorrect diacritics in the last positions of words. To evaluate the models' ability to generalize beyond observed data, we compute WER on OOV (out-of-vocabulary) words.[10]
Significance testing: We ran each experiment three times and reported the mean score.11 We used the t-test with  $p = 0.05$  to evaluate whether the difference between models' performance and the diacritic restoration is significant (Dror et al., 2018).
Table 2 shows the performance of joint diacritic restoration models when different tasks are considered. When we consider WordToChar as input to the diacritic restoration model, we observe statistically significant improvements for all evaluation metrics. This is justified by the ability of word embeddings to capture syntactic and semantic information at the sentence level. The same character is disambiguated in terms of the surrounding context
as well as the word it appears in (e.g. the character  $t$  in the word cat would be represented slightly different than  $t$  in a related word cats or even a different word table). We consider both character based model as well as WordToChar based model as our baselines (BASE).
We use WordToChar representation rather than characters for all remaining models that jointly learn more than one task. For all experiments, we observe improvements compared to both baselines across all evaluation metrics. Furthermore, all models except DIAC+SEG outperform WordToChar diacritic restoration model in terms of WER, showing the benefits of considering output distributions for the other tasks. Despite leveraging tasks focused on syntax (SYN/POS) or morpheme boundaries (SEG), the improvements extend to lexical diacritics as well. Thus, the proposed joint diacritic restoration model is also helpful in settings beyond word final syntactic related diacritics. The best performance is achieved when we consider all auxiliary tasks within the diacritic restoration model.
Impact of Auxiliary Tasks: We discuss the impact of adding each investigated task towards the performance of the diacritic restoration model.
Word segmentation (DIAC+SEG): When morpheme boundaries as well as diacritics are learned jointly, the WER performance is slightly reduced on all and OOV words. This reduction is attributed mostly to lexical diacritics. As Arabic exhibits a non-concatenative fusional morphology, reducing its complexity to a segmentation task might inherently obscure morphological processes for each form.
Observing only slight improvement is surprising; we believe that this is due to our experimental setup and does not negate the importance of having morphemes that assign the appropriate diacritics. We speculate that the reason for this is that we do not capture the interaction between morphemes as an entity, losing some level of morphological information.
For instances, the words  $\text{waham} \sim a$  versus  $\text{wahum}$  for the undiacritized words  $\text{whm}$  (bold letters refer to consonants distinguishing it from diacritics) would benefit from morpheme boundary identifications to tease apart  $\text{wa}$  from  $\text{hum}$  in the second variant  $(\text{wahum})$ , emphasizing that these are two words. But on the other hand, it adds an
10 Words that appear in the training dataset but do not appear in the test dataset. Higher number of experiments provide more robust conclusion about the models' performance. We only considered the minimum acceptable number of times to run each experiment due to limited computational resources.
8242
Task	WER	DER	LER/Lex	OOV WER
Zalmout and Habash (2017)	8.21	-	-	20.2
Zalmout and Habash (2019a)	7.50	-	-	-
Alqahtani and Diab (2019a)	7.6	2.7	-	32.1
BASE (Char)	8.51 (±0.01)	2.80	5.20/5.54	34.56
BASE (WordToChar)	8.09 (±0.05)	2.73	5.00/5.30	32.10
DIAC+SEG	8.35 (±0.02)	2.82	5.20/5.46	33.97
DIAC+SYN	7.70* (±0.02)	2.60	4.72/5.08	30.94
DIAC+POS	7.86* (±0.14)	2.65	4.72/5.20	32.28
DIAC+SEG+SYN	7.70* (±0.05)	2.59	4.65/5.03	31.33
DIAC+SEG+POS	7.73* (±0.08)	2.62	4.73/5.01	31.31
DIAC+SYN+POS	7.72* (±0.06)	2.61	4.62/5.06	31.05
ALL	7.51* (±0.09)	2.54	4.54/4.91	31.07
Table 2: Performance of the joint diacritic restoration model when different related tasks are considered. Bold numbers represent the highest score per column. Almost all scores are higher than the base model BASE (char). * denotes statistically significant improvements compared to the baselines. Lex refers to the percentage of words that have incorrect lexical diacritics only, excluding syntactic diacritics.
additional layer of ambiguity for other cases like the morpheme ktb in the diacritic variants kataba, kutubu, sayakotubo - note that the underlined segment has the same consonants as the other variants - in which identifying morphemes increased the number of possible diacritic variants without learning the interactions between adjacent morphemes.
Furthermore, we found inconsistencies in the dataset for morphemes which might cause the drop in performance when we only consider SEG. When we consider all tasks together, these inconsistencies are reduced because of the combined information from different linguistic signals towards improving the performance of the diacritic restoration model.
Syntactic diacritization (DIAC+SYN): By enforcing inflectional diacritics through an additional focused layer within the diacritic restoration model, we observe improvements on WER compared to the baselines. We notice improvements on syntactic related diacritics (LER score), which is expected given the nature of syntactic diacritization in which it learns the underlying syntactic structure to assign the appropriate syntactic diacritics for each word. Improvements also extend to lexical diacritics, and this is because word relationships are captured during learning syntactic diacritics in which BiLSTM modeling for words is integrated.
POS tagging (DIAC+POS): When we jointly train POS tagging with full diacritic restoration, we notice improvements compared to both baselines. Compared to syntactic diacritization, we obtain similar findings across all evaluation metrics except for WER on OOV words in which POS
tagging drops. Including POS tagging within diacritic restoration also captures important information about the words; the idea of POS tagging is to learn the underlying syntax of the sentence. In comparison to syntactic diacritization, it involves different types of information like passivization which could be essential in learning correct diacritics.