SlowGuess/ACL_2020_OCR huggingface dataset:
Nasser Zalmout and Nizar Habash
Computational Approaches to Modeling Language Lab
New York University Abu Dhabi
{nasser.zalmout,nizar.habash}@nyu.edu
The written forms of Semitic languages are both highly ambiguous and morphologically rich: a word can have multiple interpretations and is one of many inflected forms of the same concept or lemma. This is further exacerbated for dialectal content, which is more prone to noise and lacks a standard orthography. The morphological features can be lexicalized, like lemmas and diacritized forms, or non-lexicalized, like gender, number, and part-of-speech tags, among others. Joint modeling of the lexicalized and non-lexicalized features can identify more intricate morphological patterns, which provide better context modeling, and further disambiguate ambiguous lexical choices. However, the different modeling granularity can make joint modeling more difficult. Our approach models the different features jointly, whether lexicalized (on the character-level), or non-lexicalized (on the word-level). We use Arabic as a test case, and achieve state-of-the-art results for Modern Standard Arabic with  $20\%$  relative error reduction, and Egyptian Arabic with  $11\%$  relative error reduction.
Morphological modeling in Semitic languages is challenging. Their optional short vowels (diacritics) increase the overall ambiguity of surface forms; and their morphological richness results in large target spaces, which increase model sparsity. The different morphological features can be modeled through combined feature tags, using a single (but very large) target space, or through having separate models for each of the features. The combined features approach models the relationships between the different features explicitly, but the large target spaces for morphologically rich languages further increase sparsity. On the other hand, separate feature modeling guarantees smaller target spaces for the individual features, but the hard separation
between the features prevents modeling any interfeature dependencies. The set of morphological features includes lexicalized and non-lexicalized features, which further exacerbates joint modeling. Non-lexicalized features, like gender, and number, among others, have limited target spaces, and usually modeled as tagging tasks. Lexicalized features, like lemmas and diacritized forms (for Semitic languages), are open-ended, with large target vocabularies. Moreover, non-lexicalized features are modeled on the word level, whereas lexicalized features are optimally modeled on the character level. This difference in the modeling granularity can be challenging for joint models.
In this paper we present a model for handling lexicalized and non-lexicalized features jointly. We use a sequence-to-sequence architecture, with different parameter sharing strategies at the encoder and decoder sides for the different features. The non-lexicalized features are handled with a tagger, which shares several parameters with the encoder, and uses a multitask-learning architecture to model the different non-lexicalized features jointly. The lexicalized features, on the other hand, are handled with a specific decoder for each feature, sharing the same encoder. Our architecture models the non-lexicalized features on the word level, with a context representation that spans the entire sentence. The lexicalized features are modeled on the character level, with a fixed character context window. The character level modeling is also suitable for surface form normalization, which is important for noisy texts common in dialectal content.
We use Modern Standard Arabic (MSA) and Egyptian Arabic (EGY) as test cases. Our joint model achieves  $20\%$  relative error reduction  $(1.9\%)$  absolute improvement for MSA, and  $11\%$  relative error reduction  $(2.5\%)$  absolute improvement for EGY, compared to a baseline that models the different morphological features separately.
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8297-8307 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
The rest of the paper is structured as follows. We present a brief background and a survey of related work in Section 2. We introduce the approach and various models in Section 3, and discuss the experimental setup and results in Section 4. We conclude and provide some directions for future work in Section 5.
In this section we present a brief linguistic overview of the challenges facing morphological modeling in Semitic and morphologically rich languages. We then discuss related contributions in literature, and how our model compares to them.
2.1 Linguistic Introduction
Morphologically rich languages (MRLs) tend to have more fully inflected words than other languages, realized through many morphemes that represent several morphological features. The target space for the combined morphological features therefore tends to be large, which increases sparsity. MRLs also can be highly ambiguous, with different interpretations of the same surface forms. Ambiguity is further exacerbated for Semitic languages, like Arabic and Hebrew, at which the short vowels (diacritics) can be kept or dropped. The high degree of ambiguity in Arabic results in having about 12 analyses per word on average (Pasha et al., 2014).<sup>1</sup>
Both morphological richness and ambiguity can be modeled with morphological analyzers, or morphological dictionaries, which are used to encode all potential word inflections in the language. Morphological analyzers should ideally return all the possible analyses of a surface word (to model ambiguity), and cover all the inflected forms of a word lemma (to model morphological richness), covering all related features. The best analysis can then be chosen through morphological disambiguation; by predicting the different morphological feature values and use them to rank the relevant analyses from the analyzer. The morphological features that we model for Arabic include:
- Lexicalized features: lemmas (lex) and dia-critized forms (diac).
- Non-lexicalized features: aspect (asp), case (cas), gender (gen), person (per), part-of
speech (POS), number (num), mood (mod), state (stt), voice (vox).
- Clitics: enclitics, like pronominal enclitics, negative particle enclitics; proclitics, like article proclitic, preposition proclitics, conjunction proclitics, question proclitics.
Table 1 shows an example highlighting the different morphological features. The example presents a subset of the possible analyses for the word  $\rho$  -  $\downarrow$  lmthm.  ${}^{2}$  Disambiguation using the non-lexicalized features only is not conclusive enough,as we see in the last two analyses,where the lemma and diacriti-zation only can disambiguate the right analysis.
Dialectal Arabic (DA) includes several dialects of Arabic, like EGY, that vary by the geographical location in the Arab world. DA is also Semitic and an MRL, but it is mainly spoken, and lacks a standard orthography (Habash et al., 2012a). The lack of a standard orthography further increases sparsity and ambiguity, hence requiring explicit normalization. Habash et al. (2012a, 2018) proposed CODA, a Conventional Orthography for Dialectal Arabic, which aims to provide a conventionalized orthography across the various Arabic dialects. We use CODA as the reference for the normalization task.
2.2 Morphological Tagging
Arabic morphological tagging and disambiguation have been studied extensively in literature, with contributions for MSA (Khalifa et al., 2016; Abdelali et al., 2016; Habash and Rambow, 2005; Diab et al., 2004), and DA (Habash et al., 2013; Al-Sabbagh and Girju, 2012; Duh and Kirchhoff, 2005). There are also several recent contributions that showed significant accuracy improvement using deep learning models (Zalmout et al., 2018; Inoue et al., 2017; Zalmout and Habash, 2017; Heigold et al., 2016). In addition to other deep learning contributions that showed limited success for Arabic (Shen et al., 2016). Most of these contributions model the different morphological features separately, or focus on a limited feature subset. We elaborate on the contributions with some joint modeling aspects later in the section.
2.3 Diacritization and Lemmatization
Diacritization and lemmatization are very useful for tasks like information retrieval, machine translation, and text-to-speech, among others.
<sup>1</sup>For more information on Arabic natural language processing, see (Habash, 2010).
2Arabic transliteration is presented in the Habash-Soudi-Buckwalter scheme (Habash et al., 2007).
8298
Diacrtization	Lemma	English	POS	Prc3	Prc2	Prc1	Prc0	Per	Asp	Vox	Mod	Gen	Num	Stt	Cas	Enc0
lam~at.hum	lam~	she collected them	verb	0	0	0	0	3	p	a	i	f	s	na	na	\( \text{dobj}_{3\mathrm{{mp}}} \)
lum.tahum	lAm	you [m.s.] blamed them	verb	0	0	0	0	2	p	a	i	m	s	na	na	\( \text{dobj}_{3\mathrm{{mp}}} \)
lum.tihim	lAm	you [f.s.] blamed them	verb	0	0	0	0	2	p	a	i	f	s	na	na	\( \text{dobj}_{3\mathrm{{mp}}} \)
lum.tuhum	lAm	I blamed them	verb	0	0	0	0	1	p	a	i	m	s	na	na	\( \text{dobj}_{3\mathrm{{mp}}} \)
lam~atuhum	lam~a\( \hbar \)	their collection	noun	0	0	0	0	na	na	na	na	f	s	c	n	\( \text{poss}_{3\mathrm{{mp}}} \)
limut~ahamī	mut~aham	for a suspect	noun	0	0	li (prep)	0	na	na	na	na	m	s	i	g	0
limut~ahimī	mut~ahim	for an accuser	noun	0	0	li (prep)	0	na	na	na	na	m	s	i	g	0
Table 1: A subset of all the possible analyses for the word  $\left( {m + 1}\right) {lmthm}$  . Notice that in the last two analyses the words are disambiguated through the lemmas and diacritized forms only, and they share all the other features.
Diacritization has generally been an active area of research (Darwish et al., 2017; Zitouni et al., 2006; Nelken and Shieber, 2005). More recent contributions use Deep Learning models in different configurations; Belinkov and Glass (2015) model diacritization as a classification task, using Long Short Term Memory (LSTM) cells. And Abandah et al. (2015) use LSTMs to model diacritization as a sequence transcription task, similar to Mubarak et al. (2019) who model diacritization as a sequence-to-sequence task.
Early contributions for lemmatization used finite state machines (Schmid et al., 2004; Minnen et al., 2001), which had a limited capacity for modeling unseen words or lemmas. There were also several contributions that utilize a joint tagging and lemmatization approach, using CRFs and Maximum Entropy models (Müller et al., 2015; Chrupala et al., 2008). Other contributions approached lemmatization as a lemma selection task (Ezeiza et al., 1998), where the goal is to select the correct lemma from a set of lemmas provided by a morphological analyzer. Many of the lemmatization models for Arabic use a similar approach (Pasha et al., 2014; Roth et al., 2008). More recently, sequence-to-sequence models with attention (Bahdanau et al., 2014) have been shown useful in several NLP tasks, with several lemmatization contributions (Malaviya et al., 2019; Bergmanis and Goldwater, 2018; Pütz et al., 2018). Other contributions use additional morphosyntactic features as part of the modeling architecture (Kanerva et al., 2019; Kondratyuk et al., 2018), somewhat similar to our approach.
2.4 Joint Morphological Modeling in Arabic
There are also several contributions for the joint modeling of the different morphological features in Arabic. However, most of these contributions use separate models for each of the features, and usually use a ranking step to select the best overall morphological analysis from an external morphological
analyzer (Roth et al., 2008; Habash and Rambow, 2007). MADAMIRA (Pasha et al., 2014) is a popular system for Arabic morphological tagging and disambiguation. It uses SVMs for the different non-lexicalized features, and n-gram language models for the lemmas and diacritized forms. Zalmout and Habash (2017) presented a neural extension of this model, with LSTM taggers for the individual features, and neural language models for the lexicalized features. Inoue et al. (2017) used multi-task learning for fine-grained POS tagging, modeling the different morphological features jointly, but they do not model lemmas or diacritized forms. Zalmout and Habash (2019) also used multitask learning for the different non-lexicalized morphological features, and neural language models for lemmas and diacritized forms. This model currently provides state-of-the-art results for Arabic. In the models that rely on morphological analyzers (Zalmout and Habash, 2019, 2017; Pasha et al., 2014) surface form normalization are byproducts of selecting the correct analysis, rather than being explicitly modeled.
Non-lexicalized features are usually modeled on the word level, whereas lexicalized features are better handled through character level models. Moreover, the context representation for morphological tagging of the non-lexicalized features usually spans the entire sentence, using LSTMs for example. The optimal context representation for the lexicalized features, on the other hand, is through a fixed number of characters before and after the target word (Bergmanis and Goldwater, 2018). This difference in modeling granularity, in terms of context representation or word/character level modeling, can be very challenging for joint modeling.
We use a modified sequence-to-sequence architecture, where some components of the encoder are shared between a tagger, for the non-lexicalized
8299
features, and the encoder-decoder architecture, for the lexicalized features. We also use separate decoders for the different lexicalized features, that share the same encoder and trained jointly using a shared loss function. The remainder of this section discusses the architecture in more detail.
3.1 Tagger
The tagging architecture is similar to the architecture presented by Zalmout and Habash (2019). We use two Bi-LSTM layers on the word level to model the context for each direction of the target word. The context in the tagging network spans the entire input sentence. For each sentence of length  $L$ $\{w_{1},w_{2},\dots,w_{L}\}$ , every word  $w_{j}$  is represented by vector  $\mathbf{v}_j$ , which is comprised of the concatenation:  $\mathbf{v}_j = [\mathbf{w}_j;\mathbf{s}_j;\mathbf{a}_j]$ , where  $\mathbf{w}_j$  is the word embedding vector,  $\mathbf{s}_j$  is a vector representation of the characters within the word, and  $\mathbf{a}_j$  is a vector representing all the candidate morphological tags (from an analyzer), for all the non-lexicalized morphological features.
To obtain the vector  $\mathbf{s}_j$ , we use an LSTM-based model, applied to the character sequence in each word separately. We use the last state vector as the embedding representation of the word's characters. Whereas to get the  $\mathbf{a}_j$  vector, for each morphological feature  $f$ , we use a morphological analyzer to obtain all possible feature values of the word to be analyzed. We then embed each value separately (with separate embedding tensors for each feature, learnt within the model), then sum all the resulting vectors to get  $\mathbf{a}_j^f$  (since these tags are alternatives and do not constitute a sequence) (Zalmout and Habash, 2019). We concatenate the individual  $\mathbf{a}_j^f$  vectors for each morphological feature  $f$  of each word, to get a single representation,  $\mathbf{a}_j$ , for all the features:
$$
\mathbf {a} _ {j} ^ {f} = \sum_ {n = 1} ^ {N _ {f}} \mathbf {a} _ {j, n} ^ {f}
$$
$$
\mathbf {a} _ {j} = [ \mathbf {a} _ {j} ^ {p o s}; \dots ; \mathbf {a} _ {j} ^ {n u m}; \dots ; \mathbf {a} _ {j} ^ {v o x} ]
$$
Where  $N_{f}$  is the set of possible candidate values for each feature  $f$  (from the analyzer). The  $\mathbf{a}_j$  vector does not constitute a hard constraint and can be discarded if a morphological analyzer is not used.
Several previous contributions for Arabic showed that pretraining the word embeddings is very useful (Erdmann et al., 2018; Watson et al., 2018; Zalmout and Habash, 2017), including the baselines used in this paper. We therefore pre-train
the word embeddings with FastText (Bojanowski et al., 2017), using a large external dataset. The pre-trained embeddings are fixed during the model training. The character and tag embeddings are learnt within the model.
We use a multitask learning setup to train the different morphological features jointly, through sharing the parameters of the hidden layers in the BiLSTM network. The input is also shared, through the  $\mathbf{v}_j$  vector. The output of the network is then fed to a separate non-linearity function, output layer, and softmax, for a probability distribution of each of the features separately. Figure 1 shows the overall tagging architecture.
3.2 Encoder
We share the character and word embeddings from the tagger network in the encoder. The input context is modeled through a sliding window of a fixed number of characters around the target word, as in the Lematus model (Bergmanis and Goldwater, 2018). We also use additional special symbols for the whitespace and target word boundaries. In addition to the character embeddings, we also condition on the word level embedding of the word containing the characters. We concatenate the word embedding vector with the input character embeddings. Each character embedding  $\mathbf{c}_i$  is replaced by the concatenation  $[\mathbf{c}_i; \mathbf{w}_j]$ , where  $\mathbf{w}_j$  is the  $d_w$ -dimensional word embedding of the word  $j$  in which character  $i$  appears in. Given the characters of input sentence  $c$  and its lemmatized equivalent  $y$ , the goal is to model  $P(y_k | \mathbf{c}_i, \mathbf{w}_j)$ .
We then feed the input vectors to a network of two Bi-LSTM layers for the hidden representation at the encoder.
3.3 Decoders
We use separate decoders for lemmatization and diacritization, with two LSTM layers for each. Both decoders share the same input and parameters of the encoder Bi-LSTM network. For each decoder, we condition on the decoder output of the previous step, along with Luong attention (Luong et al., 2015) over the encoder outputs  $h_i$ , and the predicted tags from the tagger. We use the last encoder output as the initial states for the decoder layers. We use scheduled sampling (Bengio et al., 2015) during training, and feed the  $d_c$ -dimensional character embeddings at every time step. But we found empirically that using a constant sampling probability instead of scheduling provides better results.
8300
We also use dropout on the non-recurrent connections of both the encoder and decoder layers during training. The decoder outputs are fed to a softmax layer that reshapes the vectors to dimension  $d_{\text{voc}}$ , then argmax to yield an output sequence  $\mathbf{y}$  one character at a time.
Conditioning on the Predicted Tags In addition to the attention distribution and the previous time step, we also condition on the predicted tags from the tagger during decoding. The goal is to provide an additional contextual signal to the decoders, and to disambiguate the possible lexical choices. We use the output of the argmax (over the softmax distribution) for each feature, and concatenate the different tags as in the  $\mathbf{a}_j$  vector:
$$
\hat {\mathbf {t}} _ {j} = [ \hat {\mathbf {t}} _ {j} ^ {a s p}; \dots ; \hat {\mathbf {t}} _ {j} ^ {p o s}; \dots ; \hat {\mathbf {t}} _ {j} ^ {v o x} ]
$$
Figure 1: The tagger model, showing the multitask learning architecture for the features. The concatenated predicted tags are used to condition on, at the decoders.
Preventing Backpropagation to Tagger The decoder produces the lexicalized features at the character level, whereas the predicted tags are on the word level. The different granularities might create some biases, and we found that backpropagating gradients from the decoder to the tagger network leads to instability at the tagger. Therefore, we prevent the decoder from backpropagating gradients to the tagger during training. This is consistent with the model of Kondratyuk et al. (2018).
3.4 Surface Form Normalization
We use the term normalization in the sense of enriched normalization introduced by El Kholy and Habash (2012) for MSA; and in the sense of spelling conventionalization (into CODA) for DA as described by Eskander et al. (2013). Both
Figure 2: The sequence-to-sequence architecture for the lexicalized features, with a shared encoder, and separate decoders for lemmatization and diacritization. The figure does not show the fixed context window of 10 characters before and after the target word.
are non-trivial tasks comparable to true-casing or spelling correction for other languages.
The normalization task is particularly important for dialectal content, which lack a standardized orthography. The training data that we use has the diacritized annotations already in the CODA normalized form for EGY. So the output sequence of the diacritization task should be both the diacritized and CODA normalized version of the input sequence. This normalization is learnt explicitly in our character level sequence-to-sequence model. For MSA there is no need for CODA normalization, so the normalized output includes any error correction that might happen in the training dataset. Normalization is assessed as part of the overall diacritization accuracy.
3.5 Training Procedure
We use a small held out tuning set of about  $5\%$  of the training data to save the best model during training. We did not use the development set here to be consistent with other contributions in literature, where the development set is primarily used to evaluate high level design decisions only. We train the model for a fixed number of epochs and select the model that performs best on the tuning set. This method provided the most stable results, compared to early stopping or other methods.
The loss function is based on minimizing cross entropy  $H$  for each feature  $f$ . The overall loss is the average of the individual losses for the different features, whether lexicalized or non-lexicalized:
8301
$$
H (\hat {y}, y) = \frac {1}{| F |} \sum_ {f \in F} H (\hat {y} ^ {f}, y ^ {f})
$$
Where  $F$  is the set of features that we model.  $y$  represents the true feature value, and  $\hat{y}$  is the predicted value. We experimented with having different optimizers for the lexicalized and non-lexicalized features. We also experimented with a weighted average for the different features, where the weights are learnt as part of the end-to-end system. None of these modifications provided any improvement. We use Adam optimizer (Kingma and Ba, 2014) with a learning rate of 0.0005, and we run the various models for 50 epochs.
3.6 Full Morphological Disambiguation
Morphological disambiguation involves predicting the right combination of morphological features for each word in context. We can either present the predicted features from the model directly, or use a morphological analyzer to guarantee more consistent feature values. If a morphological analyzer is used, the disambiguation system selects the optimal analysis for the word from the set of analyses returned by the analyzer. We use the predicted tags to rank the analyses, and select the analysis with highest number of matched feature values. The different features can be assigned different weights during ranking. Refer to other contributions that use a similar approach for more details (Zalmout and Habash, 2019, 2017; Pasha et al., 2014).
We use the Penn Arabic Treebank (PATB parts 1,2, and 3) (Maamouri et al., 2004) for MSA, and the ARZ dataset (Maamouri et al., 2012) from the Linguistic Data Consortium (LDC), parts 1-5, for EGY. We use the same datasets as used in MADAMIRA (Pasha et al., 2014), which involves synchronizing the datasets with morphological analyzers, using the process described by Habash and Rambow (2005). We follow the data splits recommended by Diab et al. (2013) for TRAIN, DEVTEST, and
BLINDTEST. Both datasets include gold annotations for the diacritized forms, lemmas, and the remaining 14 features. The diacritized forms are normalized following the CODA guidelines for EGY. We use Alif/Ya and Hamza normalization, which is commonly used for morphological modeling in Arabic (Zalmout et al., 2018; Pasha et al., 2014; Habash et al., 2013).